Oct 30 19:32:00.760186 ip-10-0-132-115 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:32:00.760198 ip-10-0-132-115 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:32:00.760208 ip-10-0-132-115 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:32:00.760563 ip-10-0-132-115 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:32:10.783745 ip-10-0-132-115 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:32:10.783760 ip-10-0-132-115 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b1a8c0b01a224cd481929fb03acad93d -- Oct 30 19:33:54.189765 ip-10-0-132-115 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:33:54.587882 ip-10-0-132-115 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:33:54.587882 ip-10-0-132-115 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:33:54.587882 ip-10-0-132-115 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:33:54.587882 ip-10-0-132-115 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:33:54.593158 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593069 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.593158 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593153 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.593158 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593158 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.593158 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593162 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.593158 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593166 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593171 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593175 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593179 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593182 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593186 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593190 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593193 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593197 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593200 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593205 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593209 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593213 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593217 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593220 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593224 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593228 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593232 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593236 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593241 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.593459 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593245 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593249 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593260 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593264 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593268 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593272 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593276 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593280 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593284 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593288 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593292 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593296 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593300 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593305 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593310 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593314 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593318 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593323 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593327 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.594326 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593333 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593339 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593343 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593348 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593352 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593357 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593362 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593366 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593370 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593374 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593379 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593384 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593389 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593393 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593398 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593402 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593408 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593412 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593416 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593420 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.595125 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593424 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593429 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593433 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593437 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593442 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593446 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593450 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593454 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593458 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593462 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593466 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593472 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593478 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593484 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593508 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593514 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593519 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593524 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593528 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.595986 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593533 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593537 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593541 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593545 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593550 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593554 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593558 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593562 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593566 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593570 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593576 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593580 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593584 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593588 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593592 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593596 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593600 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593604 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593608 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593612 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.596831 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593617 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593623 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593628 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.593632 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594932 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594943 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594948 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594952 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594957 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594962 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594967 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594972 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594977 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594981 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594985 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594989 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594993 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.594997 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595001 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.597356 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595005 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595010 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595014 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595018 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595023 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595027 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595031 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595036 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595040 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595044 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595048 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595052 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595056 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595061 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595066 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595070 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595074 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595078 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595082 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595086 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.597855 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595090 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595095 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595100 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595104 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595108 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595112 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595116 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595120 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595124 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595128 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595132 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595135 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595139 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595143 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595147 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595151 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595155 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595162 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595169 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595173 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.598738 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595178 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595182 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595186 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595191 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595195 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595199 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595204 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595209 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595213 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595218 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595222 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595226 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595230 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595234 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595238 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595242 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595246 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595251 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595255 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595259 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.599436 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595263 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595270 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595274 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595278 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595282 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595286 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595291 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595316 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595322 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595327 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595333 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595338 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595342 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595349 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595354 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595359 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595364 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595368 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595373 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.600143 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595377 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595381 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595385 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595389 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595393 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595397 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595401 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595406 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595410 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595414 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595418 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.595423 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596240 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596249 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596254 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596272 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596277 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596281 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596285 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.600832 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596289 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596294 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596298 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596303 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596308 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596313 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596318 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596322 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596327 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596331 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596335 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596339 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596343 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596347 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596352 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596356 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596361 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596365 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596370 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596374 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.601402 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596379 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596384 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596388 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596392 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596396 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596400 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596405 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596409 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596413 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596417 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596421 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596425 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596430 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596433 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596437 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596441 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596446 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596454 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596460 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596464 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.601928 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596470 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596473 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596478 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596482 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596486 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596508 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596513 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596518 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596522 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596526 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596530 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596533 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596537 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596542 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596546 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596549 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596554 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596560 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596566 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.602410 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596572 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596577 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596582 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596587 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596591 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596595 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596599 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596605 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596611 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596616 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596623 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596628 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596632 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596637 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596641 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596645 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596649 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596653 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596657 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.602884 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596662 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596666 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596670 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596674 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596678 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596682 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596686 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596690 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596694 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596698 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596704 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596708 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596712 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596716 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596720 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596724 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596729 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596733 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596737 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596741 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.603359 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.596745 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597855 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597870 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597880 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597888 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597896 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597901 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597908 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597915 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597919 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597925 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597930 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597935 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597940 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597945 2571 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597949 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597954 2571 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597959 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597963 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597970 2571 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597975 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597980 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597985 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597990 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:33:54.603849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.597996 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598001 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598005 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598011 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598015 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598020 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598024 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598029 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598034 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598041 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598045 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598050 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598055 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598059 2571 flags.go:64] FLAG: --enable-server="true" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598065 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598071 2571 flags.go:64] FLAG: --event-burst="100" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598076 2571 flags.go:64] FLAG: --event-qps="50" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598081 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598085 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598090 2571 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598096 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598102 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598107 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598111 2571 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598116 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:33:54.604504 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598121 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598126 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598130 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598135 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598140 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598144 2571 flags.go:64] FLAG: --feature-gates="" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598150 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598155 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598160 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598165 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598170 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598175 2571 flags.go:64] FLAG: --help="false" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598180 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598185 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598190 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598195 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598200 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598206 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598210 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598215 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598219 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598224 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598234 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598239 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:33:54.605126 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598243 2571 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598248 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598253 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598257 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598262 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598267 2571 flags.go:64] FLAG: --lock-file="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598272 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598277 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598281 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598290 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598295 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598299 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598304 2571 flags.go:64] FLAG: --logging-format="text" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598308 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598314 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598318 2571 flags.go:64] FLAG: --manifest-url="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598323 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598330 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598335 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598341 2571 flags.go:64] FLAG: --max-pods="110" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598346 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598351 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598355 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598360 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598365 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:33:54.605732 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598369 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598374 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598384 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598389 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598394 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598399 2571 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598404 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598409 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598414 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598418 2571 flags.go:64] FLAG: --port="10250" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598423 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598427 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0963631a1df966acd" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598432 2571 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598438 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598442 2571 flags.go:64] FLAG: --register-node="true" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598447 2571 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598457 2571 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598462 2571 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598466 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598470 2571 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598477 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598481 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598486 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598510 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598518 2571 flags.go:64] FLAG: --runonce="false" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598522 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:33:54.606322 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598527 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598532 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598536 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598541 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598545 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598550 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598555 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598559 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598564 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598568 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598573 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598578 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598582 2571 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598588 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598597 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598602 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598606 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598614 2571 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598619 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598623 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598628 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598633 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598638 2571 flags.go:64] FLAG: --v="2" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598645 2571 flags.go:64] FLAG: --version="false" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598651 2571 flags.go:64] FLAG: --vmodule="" Oct 30 19:33:54.606945 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598657 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.598663 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600954 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600970 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600974 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600979 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600982 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600985 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600988 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600991 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600994 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600997 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.600999 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601002 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601006 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601010 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601013 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601016 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601019 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601022 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.607565 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601024 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601027 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601030 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601037 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601041 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601044 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601047 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601049 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601052 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601055 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601058 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601060 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601063 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601066 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601071 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601074 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601077 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601079 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601082 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601084 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.608074 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601087 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601090 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601092 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601095 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601097 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601102 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601105 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601107 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601110 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601112 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601115 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601118 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601120 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601123 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601128 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601130 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601133 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601136 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601139 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601141 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.608568 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601144 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601146 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601149 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601152 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601154 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601157 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601163 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601166 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601168 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601171 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601173 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601176 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601179 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601182 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601184 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601187 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601189 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601192 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601194 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601197 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.609050 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601199 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601202 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601204 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601207 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601210 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601214 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601217 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601219 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601222 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601226 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601229 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601233 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601236 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601239 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601241 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601244 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601247 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601250 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601255 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.609587 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601257 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601260 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601262 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601265 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601268 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601270 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601273 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601276 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601278 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.601310 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601350 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601354 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601357 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601360 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601364 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.610032 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601368 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601371 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601374 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601376 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601380 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601382 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601385 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601388 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601390 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601393 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601396 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601399 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601401 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601404 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601407 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601410 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601414 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601417 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601420 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.610383 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601423 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601425 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601428 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601431 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601433 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601436 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601438 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601441 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601444 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601446 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601449 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601452 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601454 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601457 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601459 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601462 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601464 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601467 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601470 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601473 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.610935 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601475 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601478 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601481 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601483 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601485 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601488 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601505 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601508 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601510 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601514 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601517 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601519 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601522 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601524 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601527 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601529 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601532 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601534 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601537 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.611416 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601539 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601542 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601544 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601546 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601549 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601551 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601554 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601556 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601559 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601561 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601564 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601566 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601569 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601572 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601575 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601577 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601582 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601586 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601589 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601592 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.611883 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601594 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601597 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601599 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601602 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601604 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601607 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601609 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601612 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601614 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601617 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601619 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601622 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601624 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601627 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601629 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601632 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601634 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601637 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601639 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601642 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.612353 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601644 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601647 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.601651 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.608871 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.608887 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.608903 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.608907 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608963 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608968 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608972 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608976 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608979 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608981 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608984 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608987 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608990 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608993 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608995 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.608998 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609000 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.612829 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609003 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609006 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609009 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609011 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609014 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609017 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609019 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609023 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609027 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609029 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609032 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609035 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609037 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609040 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609042 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609045 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609049 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609052 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609055 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.613329 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609057 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609060 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609063 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609066 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609068 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609071 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609073 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609076 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609078 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609081 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609084 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609086 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609089 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609091 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609094 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609096 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609099 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609101 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609104 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609107 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.613897 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609109 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609111 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609114 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609117 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609119 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609122 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609124 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609127 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609130 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609132 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609135 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609138 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609140 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609143 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609145 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609148 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609150 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609153 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609155 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.614377 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609158 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609160 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609163 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609165 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609168 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609170 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609173 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609175 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609178 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609180 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609183 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609185 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609188 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609190 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609193 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609195 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609198 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609200 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609203 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609205 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.614837 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609207 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609210 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609213 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609216 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609219 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609221 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609224 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609226 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609229 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609231 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609234 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609236 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609239 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609241 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609245 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.615294 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.609273 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609404 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609408 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609411 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609414 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609416 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609419 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609421 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609424 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609427 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609429 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609432 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609434 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609437 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609439 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609441 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609444 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609447 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609449 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609452 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.615679 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609454 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609456 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609460 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609463 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609465 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609468 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609470 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609473 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609475 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609478 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609480 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609482 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609485 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609487 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609507 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609511 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609515 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609517 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609520 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.616135 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609522 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609525 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609527 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609530 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609532 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609535 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609537 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609540 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609542 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609545 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609548 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609550 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609553 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609555 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609558 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609560 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609563 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609566 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609569 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609571 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.616597 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609574 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609576 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609579 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609582 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609584 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609587 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609590 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609592 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609595 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609598 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609600 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609603 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609606 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609608 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609611 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609613 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609616 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609618 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609620 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609623 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609625 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.617099 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609628 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609630 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609633 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609635 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609637 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609640 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609642 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609645 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609648 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609651 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609657 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609659 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609662 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609664 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609667 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609670 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609673 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609677 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.617612 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609680 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609683 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609685 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609688 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609692 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609696 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609699 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609702 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609704 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.609728 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609761 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609765 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609768 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609772 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609774 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:33:54.618076 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609777 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609780 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609782 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609785 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609787 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609790 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609792 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609795 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609798 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609801 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609804 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609807 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609810 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609812 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609815 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609817 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609820 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609823 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609825 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609828 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:33:54.618480 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609831 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609833 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609835 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609838 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609841 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609843 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609846 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609848 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609851 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609853 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609856 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609858 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609860 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609863 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609866 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609868 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609871 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609873 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609876 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609878 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:33:54.618967 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609881 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609885 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609889 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609892 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609894 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609897 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609900 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609902 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609905 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609907 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609910 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609912 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609915 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609918 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609921 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609923 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609926 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609929 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609931 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609934 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:33:54.619426 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609936 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609938 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609941 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609943 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609946 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609948 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609951 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609953 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609956 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609958 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609961 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609963 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609966 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609969 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609974 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609976 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609979 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609981 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609984 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:33:54.619937 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609986 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609989 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609991 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609994 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.609998 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610001 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610004 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610007 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610009 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610012 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610015 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610019 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610023 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610026 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610028 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610031 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610034 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610036 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610039 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:33:54.620420 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610041 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610044 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:54.610047 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.610100 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.612589 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.613666 2571 server.go:1014] "Starting client certificate rotation" Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.613761 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.613798 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:33:54.620873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.615898 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:33:54.621752 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.621738 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:33:54.632741 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.632724 2571 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:33:54.635883 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.635867 2571 log.go:25] "Validated CRI v1 image API" Oct 30 19:33:54.636755 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.636734 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:33:54.642200 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.642174 2571 fs.go:135] Filesystem UUIDs: map[701d8e5e-d4b4-4432-9c56-6384a7ad544a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7eda3421-4538-48a9-847b-4a0b59bdd0c0:/dev/nvme0n1p3] Oct 30 19:33:54.642278 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.642193 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:33:54.647813 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.647699 2571 manager.go:217] Machine: {Timestamp:2026-10-30 19:33:54.647222101 +0000 UTC m=+0.350073564 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097216 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ad24fe900a4cc02effb48e64270d SystemUUID:ec23ad24-fe90-0a4c-c02e-ffb48e64270d BootID:b1a8c0b0-1a22-4cd4-8192-9fb03acad93d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:d8:de:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:d8:de:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:ae:48:c0:31:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:33:54.647813 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.647802 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:33:54.647934 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.647871 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:33:54.651638 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651613 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:33:54.651767 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651640 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:33:54.651809 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651769 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:33:54.651809 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651777 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:33:54.651809 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651802 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:33:54.651886 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.651814 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:33:54.653119 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.653104 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:33:54.657933 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.657919 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:33:54.658741 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.658731 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:33:54.659622 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659613 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:33:54.659656 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659625 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:33:54.659656 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659636 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:33:54.659656 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659644 2571 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:33:54.659656 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659652 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:33:54.660002 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.659988 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:33:54.662441 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.662427 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:33:54.669271 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.669259 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:33:54.670304 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670291 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:33:54.670339 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670322 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:33:54.670522 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670510 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:33:54.670558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670529 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:33:54.670558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670535 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:33:54.670558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670540 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:33:54.670558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670546 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:33:54.670558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.670551 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:33:54.671351 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671342 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:33:54.671390 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671352 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:33:54.671390 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671359 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:33:54.671390 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671367 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:33:54.671390 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671384 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:33:54.671538 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.671394 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:33:54.672986 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.672977 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:33:54.672986 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.672986 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:33:54.677058 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.677047 2571 server.go:1263] "Started kubelet" Oct 30 19:33:54.677156 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.677121 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:33:54.677250 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.677148 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:33:54.677250 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.677222 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:33:54.677909 ip-10-0-132-115 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:33:54.678804 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.678776 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:33:54.680440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.680423 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:33:54.684796 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.684779 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:33:54.685962 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.685943 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:33:54.687309 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.687294 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:33:54.687420 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.687396 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:54.693567 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.692514 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:33:54.693843 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.693682 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:33:54.693916 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.693860 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:33:54.694176 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.694161 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:33:54.694176 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.694178 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:33:54.694674 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.694655 2571 factory.go:55] Registering systemd factory Oct 30 19:33:54.694736 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.694700 2571 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:33:54.695115 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.695099 2571 factory.go:153] Registering CRI-O factory Oct 30 19:33:54.695185 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.695119 2571 factory.go:223] Registration of the crio container factory successfully Oct 30 19:33:54.695242 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.695197 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:33:54.695242 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.695213 2571 factory.go:103] Registering Raw factory Oct 30 19:33:54.695242 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.695228 2571 manager.go:1199] Started watching for new ooms in manager Oct 30 19:33:54.696369 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.696339 2571 manager.go:319] Starting recovery of all containers Oct 30 19:33:54.699436 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.699409 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:33:54.704415 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.704394 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:33:54.705559 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.704398 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18e365a4b31358b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-10-30 19:33:54.677020856 +0000 UTC m=+0.379872319,LastTimestamp:2026-10-30 19:33:54.677020856 +0000 UTC m=+0.379872319,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Oct 30 19:33:54.706964 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.706948 2571 manager.go:324] Recovery completed Oct 30 19:33:54.710777 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.710766 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.712243 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.712227 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.712310 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.712268 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.712310 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.712280 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.713718 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.713416 2571 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:33:54.713718 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.713431 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:33:54.713718 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.713443 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.713718 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.713451 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:33:54.714301 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.714283 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.714424 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.714308 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.714424 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.714321 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.715740 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.715728 2571 policy_none.go:50] "Start" Oct 30 19:33:54.715797 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.715743 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:33:54.715797 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.715753 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:33:54.717806 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.717793 2571 policy_none.go:44] "Start" Oct 30 19:33:54.723559 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.723532 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:33:54.723652 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.723533 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:33:54.723652 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.723533 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:33:54.736956 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.736868 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18e365a4b52cc94d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-10-30 19:33:54.712242509 +0000 UTC m=+0.415093971,LastTimestamp:2026-10-30 19:33:54.712242509 +0000 UTC m=+0.415093971,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Oct 30 19:33:54.737283 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.737258 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:33:54.743130 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.743097 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdbkx" Oct 30 19:33:54.751442 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.750580 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18e365a4b52d4443 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-10-30 19:33:54.712273987 +0000 UTC m=+0.415125448,LastTimestamp:2026-10-30 19:33:54.712273987 +0000 UTC m=+0.415125448,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Oct 30 19:33:54.754025 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.754005 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdbkx" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.761250 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.762582 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18e365a4b52d6b42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-115.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-10-30 19:33:54.71228397 +0000 UTC m=+0.415135431,LastTimestamp:2026-10-30 19:33:54.71228397 +0000 UTC m=+0.415135431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764136 2571 manager.go:341] "Starting Device Plugin manager" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.764167 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764180 2571 server.go:87] "Starting device plugin registration server" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764431 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764443 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764644 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764725 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.764734 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.765206 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.765259 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.770774 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.772091 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.772111 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.772127 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:33:54.776217 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.772159 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:33:54.786268 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.786251 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:33:54.864804 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.864751 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.865638 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.865620 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.865739 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.865654 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.865739 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.865676 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.865739 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.865699 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.872904 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.872888 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal"] Oct 30 19:33:54.872948 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.872939 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.873730 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.873711 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.873814 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.873746 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.873814 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.873763 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.875972 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.875960 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.876137 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876123 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.876173 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876153 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.876702 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876689 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.876702 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876696 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.876811 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876713 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.876811 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876721 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.876811 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876739 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.876811 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.876723 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.878864 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.878851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.878906 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.878877 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:54.879576 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.879559 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:54.879635 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.879584 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:54.879635 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.879593 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:54.882183 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.882167 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.882269 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.882191 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-115.ec2.internal\": node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:54.894981 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.894953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.900917 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.900899 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-115.ec2.internal\" not found" node="ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.902871 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.902856 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-115.ec2.internal\" not found" node="ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.916948 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:54.916927 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:54.995857 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.995829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.995998 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.995865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.995998 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.995882 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:54.995998 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:54.995943 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.017288 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.017267 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.096957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.096932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.097063 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.096972 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.097063 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.097015 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.097063 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.097030 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.117569 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.117518 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.206219 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.206192 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.208140 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.208124 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:55.218618 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.218484 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.319716 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.319678 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.420340 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.420311 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.520981 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.520947 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.613606 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.613577 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:33:55.614073 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.613715 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:33:55.621883 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.621859 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.722596 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.722555 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.760885 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.760843 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:28:55 +0000 UTC" deadline="2028-04-26 23:36:00.374257087 +0000 UTC" Oct 30 19:33:55.761027 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.760880 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13060h2m4.613380541s" Oct 30 19:33:55.823509 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.823471 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:55.847919 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:55.847878 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb73836ec8ec4c4107421909743aa5c.slice/crio-913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320 WatchSource:0}: Error finding container 913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320: Status 404 returned error can't find the container with id 913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320 Oct 30 19:33:55.848305 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:55.848281 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47397ab791596d97226855f409f94517.slice/crio-86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543 WatchSource:0}: Error finding container 86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543: Status 404 returned error can't find the container with id 86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543 Oct 30 19:33:55.853929 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.853914 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:33:55.916021 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.915976 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:33:55.917210 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.917190 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:33:55.917317 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.917229 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:33:55.917317 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:55.917245 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:33:55.924374 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:55.924356 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:56.025114 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.025046 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:33:56.089146 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.089123 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:33:56.125180 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.125152 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:33:56.187508 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.187459 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:33:56.199598 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.199575 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:33:56.219166 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.219145 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:33:56.220223 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.220210 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:33:56.244608 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.244585 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:33:56.660760 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.660724 2571 apiserver.go:52] "Watching apiserver" Oct 30 19:33:56.673326 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.673294 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:33:56.687007 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.686967 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:33:56.688979 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.688959 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:33:56.689301 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.689283 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-k7rbb","kube-system/konnectivity-agent-6bzkf","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal","openshift-network-diagnostics/network-check-target-8f2fj","openshift-ovn-kubernetes/ovnkube-node-pfvgl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv","openshift-cluster-node-tuning-operator/tuned-qzmns","openshift-dns/node-resolver-jw9s8","openshift-multus/multus-l5hwp","kube-system/global-pull-secret-syncer-shwzj","openshift-image-registry/node-ca-fr4xc","openshift-multus/network-metrics-daemon-57wdt","openshift-network-operator/iptables-alerter-przhx"] Oct 30 19:33:56.691928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.691906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.694918 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.694890 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.698849 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.698533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.703404 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.703383 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.705469 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.705448 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:56.705608 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.705581 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:33:56.707284 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707403 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707280 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:33:56.707403 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707403 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707417 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707427 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\"" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7gn\" (UniqueName: \"kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707555 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.707592 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707586 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707649 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707706 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707736 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtr9\" (UniqueName: \"kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.707928 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707928 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.708368 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.708368 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.707979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.708368 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.708000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.708368 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.708253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.710516 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.710199 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.712426 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.712406 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713214 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713417 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713444 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713509 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713599 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713717 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713793 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713855 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.713953 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.713924 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\"" Oct 30 19:33:56.717051 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.716678 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.717669 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.717651 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\"" Oct 30 19:33:56.717806 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.717791 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.717865 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.717840 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.717922 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.717904 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:33:56.719534 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.719515 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:56.719639 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.719576 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:33:56.721982 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.721963 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.722087 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.722044 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:33:56.724992 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.724961 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.734555 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.734532 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:33:56.737243 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.737220 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hz6vw" Oct 30 19:33:56.749361 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.749341 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hz6vw" Oct 30 19:33:56.777463 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.777413 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerStarted","Data":"86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543"} Oct 30 19:33:56.778729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.778702 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"4bb73836ec8ec4c4107421909743aa5c","Type":"ContainerStarted","Data":"913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320"} Oct 30 19:33:56.795149 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.795124 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:33:56.808723 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808693 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.808867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:56.808867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808761 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.808867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.809040 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808882 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.809040 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.809040 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.808998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809040 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7gn\" (UniqueName: \"kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809066 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809109 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpv22\" (UniqueName: \"kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.809254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809220 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8dh\" (UniqueName: \"kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809435 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.809587 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809610 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809621 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809667 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809694 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809734 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809844 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809873 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809929 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.809997 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810061 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsfb\" (UniqueName: \"kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6t6g\" (UniqueName: \"kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810379 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810409 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810634 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810667 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810668 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810751 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.810913 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810821 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810839 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810847 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.810985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811055 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811140 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811181 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811222 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.811780 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811465 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811480 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llllb\" (UniqueName: \"kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811683 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811695 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811652 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811764 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811896 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.812633 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811923 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttzf\" (UniqueName: \"kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811958 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.811995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtr9\" (UniqueName: \"kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812081 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812117 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkknw\" (UniqueName: \"kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812288 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812334 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812372 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812550 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812597 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812639 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812745 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812844 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812869 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812892 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.812979 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.813365 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:33:56.813519 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.813425 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.813787 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.813810 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.814175 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.814247 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.814313 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.814429 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\"" Oct 30 19:33:56.814619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.814471 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\"" Oct 30 19:33:56.815538 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.815512 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:56.832315 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.832188 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7gn\" (UniqueName: \"kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:33:56.834266 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.834240 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtr9\" (UniqueName: \"kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:56.913376 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.913376 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913376 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913434 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913466 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913602 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913666 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.913706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llllb\" (UniqueName: \"kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.913995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttzf\" (UniqueName: \"kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914081 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkknw\" (UniqueName: \"kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.914411 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914161 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914198 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpv22\" (UniqueName: \"kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914390 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8dh\" (UniqueName: \"kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914473 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915106 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914662 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914717 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914717 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914734 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914784 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914806 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.914866 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914881 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914922 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.915766 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.914936 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:33:57.414914072 +0000 UTC m=+3.117765546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.914989 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915092 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915185 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915283 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915288 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915326 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915284 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915422 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915449 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915501 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915535 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915548 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915551 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.916591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915628 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915049 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915782 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915881 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsfb\" (UniqueName: \"kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6t6g\" (UniqueName: \"kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915925 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915947 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915962 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916045 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916109 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916122 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916124 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916167 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916226 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916234 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.915295 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.917418 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.916330 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.916402 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:33:57.416385068 +0000 UTC m=+3.119236520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916417 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.916430 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.917820 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.918048 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.918205 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.918392 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.918362 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.918826 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.918486 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.919061 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.919037 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.920088 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.920053 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.935309 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.935281 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:33:56.935309 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.935311 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:33:56.935497 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.935325 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:56.935497 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:56.935412 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:57.435393503 +0000 UTC m=+3.138244977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:56.938697 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.938626 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttzf\" (UniqueName: \"kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:56.939351 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.939319 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpv22\" (UniqueName: \"kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:56.940191 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.940153 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkknw\" (UniqueName: \"kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:56.940290 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.940268 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llllb\" (UniqueName: \"kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:56.941677 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.941656 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsfb\" (UniqueName: \"kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:56.942520 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.942477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6t6g\" (UniqueName: \"kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:56.942836 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:56.942813 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8dh\" (UniqueName: \"kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:57.009682 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.009639 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5hwp" Oct 30 19:33:57.018132 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.017958 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:33:57.022475 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.022448 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8699c88_8f03_4a7b_90f7_152a0118d517.slice/crio-d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf WatchSource:0}: Error finding container d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf: Status 404 returned error can't find the container with id d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf Oct 30 19:33:57.028351 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.028312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:33:57.029422 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.029389 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a17b98_bb85_4bde_b1b4_29d1b39ad1c5.slice/crio-77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7 WatchSource:0}: Error finding container 77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7: Status 404 returned error can't find the container with id 77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7 Oct 30 19:33:57.034623 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.034601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:33:57.040178 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.040152 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c799558_73c8_45a2_83bf_dd99b24b8a36.slice/crio-8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7 WatchSource:0}: Error finding container 8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7: Status 404 returned error can't find the container with id 8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7 Oct 30 19:33:57.049091 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.049060 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc913d7_1df1_409d_a0bd_9a13b9d4ab1e.slice/crio-869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99 WatchSource:0}: Error finding container 869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99: Status 404 returned error can't find the container with id 869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99 Oct 30 19:33:57.059449 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.059273 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:33:57.061607 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.061583 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:33:57.069533 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.069485 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:33:57.071874 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.071845 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcadcb7_1c64_4956_bf5c_489946be6487.slice/crio-8d935b55aaea8e981af4d2871db31027ac43d81070cb895466ce03252c64d623 WatchSource:0}: Error finding container 8d935b55aaea8e981af4d2871db31027ac43d81070cb895466ce03252c64d623: Status 404 returned error can't find the container with id 8d935b55aaea8e981af4d2871db31027ac43d81070cb895466ce03252c64d623 Oct 30 19:33:57.076092 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.076056 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c85830_f826_4df6_a48f_372c870e653a.slice/crio-79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87 WatchSource:0}: Error finding container 79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87: Status 404 returned error can't find the container with id 79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87 Oct 30 19:33:57.077896 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.077872 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:33:57.081483 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.081451 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3ab265_3e11_41d3_9192_e40c7588e1f5.slice/crio-6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd WatchSource:0}: Error finding container 6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd: Status 404 returned error can't find the container with id 6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd Oct 30 19:33:57.089405 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.089376 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cde5a04_1c75_44b3_8e49_bda52bb78768.slice/crio-c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79 WatchSource:0}: Error finding container c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79: Status 404 returned error can't find the container with id c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79 Oct 30 19:33:57.090724 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.090701 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:33:57.101836 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:33:57.101774 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded54fa06_9be0_44dc_bf64_f48ea87b081f.slice/crio-54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d WatchSource:0}: Error finding container 54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d: Status 404 returned error can't find the container with id 54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d Oct 30 19:33:57.420123 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.420089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:57.420283 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.420131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:57.420283 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.420249 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:57.420283 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.420264 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:57.420404 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.420313 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.420294863 +0000 UTC m=+4.123146315 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:57.420404 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.420334 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.420322758 +0000 UTC m=+4.123174212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:57.521333 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.520784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:57.521333 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.520980 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:33:57.521333 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.521006 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:33:57.521333 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.521020 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:57.521333 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:57.521084 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:58.521063109 +0000 UTC m=+4.223914563 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:57.752782 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.752596 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-04-25 07:27:51.637606497 +0000 UTC" Oct 30 19:33:57.752782 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.752652 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13019h53m53.884958827s" Oct 30 19:33:57.787706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.787422 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87"} Oct 30 19:33:57.793642 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.792848 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"4bb73836ec8ec4c4107421909743aa5c","Type":"ContainerStarted","Data":"6aefa0d98003385c45ed51eb15da5589dafa8e2fdc5c005ff6fbcb760b44229d"} Oct 30 19:33:57.798811 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.798725 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qzmns" event={"ID":"9dcadcb7-1c64-4956-bf5c-489946be6487","Type":"ContainerStarted","Data":"8d935b55aaea8e981af4d2871db31027ac43d81070cb895466ce03252c64d623"} Oct 30 19:33:57.811076 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.811002 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-przhx" event={"ID":"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e","Type":"ContainerStarted","Data":"869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99"} Oct 30 19:33:57.826206 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.826170 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bzkf" event={"ID":"6c799558-73c8-45a2-83bf-dd99b24b8a36","Type":"ContainerStarted","Data":"8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7"} Oct 30 19:33:57.830474 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.830443 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fr4xc" event={"ID":"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5","Type":"ContainerStarted","Data":"77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7"} Oct 30 19:33:57.833412 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.833357 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5hwp" event={"ID":"e8699c88-8f03-4a7b-90f7-152a0118d517","Type":"ContainerStarted","Data":"d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf"} Oct 30 19:33:57.844439 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.844402 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerStarted","Data":"54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d"} Oct 30 19:33:57.852848 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.852815 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jw9s8" event={"ID":"7cde5a04-1c75-44b3-8e49-bda52bb78768","Type":"ContainerStarted","Data":"c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79"} Oct 30 19:33:57.858583 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:57.858546 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" event={"ID":"ea3ab265-3e11-41d3-9192-e40c7588e1f5","Type":"ContainerStarted","Data":"6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd"} Oct 30 19:33:58.429535 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.429243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:58.429745 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.429544 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:58.429745 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.429586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:58.429745 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.429629 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:00.429606863 +0000 UTC m=+6.132458337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:33:58.429745 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.429695 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:58.429988 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.429751 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:00.429734703 +0000 UTC m=+6.132586157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:33:58.530183 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.530140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:58.530457 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.530433 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:33:58.530527 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.530471 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:33:58.530527 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.530486 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:58.530677 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.530564 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:00.530530253 +0000 UTC m=+6.233381714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:33:58.753703 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.753604 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-07-13 09:43:14.183024212 +0000 UTC" Oct 30 19:33:58.753703 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.753674 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14918h9m15.429377023s" Oct 30 19:33:58.778359 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.778335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:33:58.778550 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.778470 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:33:58.778943 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.778922 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:33:58.779048 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.779033 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:33:58.779239 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.779107 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:33:58.779239 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:33:58.779195 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:33:58.868196 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.868156 2571 generic.go:356] "Generic (PLEG): container finished" podID="47397ab791596d97226855f409f94517" containerID="4fb178cb2f35da6ca3652d465c0ed33cad4adc85cc4be13d74a644de0656b989" exitCode=0 Oct 30 19:33:58.868395 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.868270 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerDied","Data":"4fb178cb2f35da6ca3652d465c0ed33cad4adc85cc4be13d74a644de0656b989"} Oct 30 19:33:58.911772 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:58.910973 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" podStartSLOduration=2.910953206 podStartE2EDuration="2.910953206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:57.836748628 +0000 UTC m=+3.539600104" watchObservedRunningTime="2026-10-30 19:33:58.910953206 +0000 UTC m=+4.613804682" Oct 30 19:33:59.882023 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:59.881238 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerStarted","Data":"3cf8e869ec2131b7ff082c9af5a124b213e7953c40fb35c2a02e4559520bb718"} Oct 30 19:33:59.924900 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:33:59.924376 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" podStartSLOduration=3.924358153 podStartE2EDuration="3.924358153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:59.923868974 +0000 UTC m=+5.626720449" watchObservedRunningTime="2026-10-30 19:33:59.924358153 +0000 UTC m=+5.627209628" Oct 30 19:34:00.447942 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.447845 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:00.447942 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.447909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:00.448184 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.448059 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:00.448184 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.448126 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:04.448107263 +0000 UTC m=+10.150958733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:00.448307 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.448206 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:00.448307 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.448252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:04.448237541 +0000 UTC m=+10.151089008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:00.549289 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.549244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:00.549494 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.549419 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:00.549494 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.549439 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:00.549494 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.549452 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:00.549680 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.549513 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:04.549493779 +0000 UTC m=+10.252345245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:00.778088 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.778006 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:00.778278 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.778141 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:00.778559 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.778535 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:00.778700 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.778678 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:00.778766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.778749 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:00.778864 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:00.778821 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:00.890449 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.890357 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="41dcdd09319759ad0198ae4cf8863295945614d36a7e4ad739877fa3e86c25dd" exitCode=0 Oct 30 19:34:00.890970 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:00.890599 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"41dcdd09319759ad0198ae4cf8863295945614d36a7e4ad739877fa3e86c25dd"} Oct 30 19:34:01.902983 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:01.902942 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bzkf" event={"ID":"6c799558-73c8-45a2-83bf-dd99b24b8a36","Type":"ContainerStarted","Data":"beaf5b027b217ad8af0a8aa6841dcd5afac355d08b02a5ddc0038e33cab27edb"} Oct 30 19:34:01.947547 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:01.947482 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6bzkf" podStartSLOduration=2.9974297 podStartE2EDuration="6.947460255s" totalImagesPullingTime="3.950030555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:01.946810687 +0000 UTC m=+7.649662164" watchObservedRunningTime="2026-10-30 19:34:01.947460255 +0000 UTC m=+7.650311735" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:02.773254 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:02.773425 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:02.773874 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:02.773993 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:02.774052 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:02.776960 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:02.774140 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:02.910956 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:02.910599 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fr4xc" event={"ID":"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5","Type":"ContainerStarted","Data":"4dd596ba8c6aa1ed351703dad5aee86bdc68746845b9b863fd7012ced1029e04"} Oct 30 19:34:03.917160 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:03.917125 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" event={"ID":"ea3ab265-3e11-41d3-9192-e40c7588e1f5","Type":"ContainerStarted","Data":"3d3ec705445e94fef4519e3b061a5688fef37076191f59a00cb9d8f56c0c6c58"} Oct 30 19:34:04.481975 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.481892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:04.481975 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.481965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:04.482207 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.482058 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:04.482207 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.482058 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:04.482207 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.482125 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:12.482106301 +0000 UTC m=+18.184957752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:04.482207 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.482143 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:12.482133863 +0000 UTC m=+18.184985313 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:04.583236 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.583195 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:04.583503 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.583477 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:04.583503 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.583509 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:04.583503 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.583524 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:04.583749 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.583613 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:12.583593779 +0000 UTC m=+18.286445233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.772719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.772860 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.773322 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.773414 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.773486 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:04.773690 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:04.773599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:04.797887 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.797287 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:04.798969 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.798806 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:04.845076 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.845008 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fr4xc" podStartSLOduration=4.8289310069999996 podStartE2EDuration="9.844987295s" totalImagesPullingTime="5.016056288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:02.954563034 +0000 UTC m=+8.657414510" watchObservedRunningTime="2026-10-30 19:34:04.844987295 +0000 UTC m=+10.547838767" Oct 30 19:34:04.878807 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.878760 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:34:04.925248 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.925214 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" event={"ID":"ea3ab265-3e11-41d3-9192-e40c7588e1f5","Type":"ContainerStarted","Data":"a9289d9da0ec03de32eaae6b92d7b4174b9c0cc09521ad719ed63b61259a2229"} Oct 30 19:34:04.925918 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.925270 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:04.926123 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:04.926103 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:05.770072 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.769200 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:34:04.878800368Z","UUID":"657629ec-1fdd-4f55-86de-d40c3fc45561","Handler":null,"Name":"","Endpoint":""} Oct 30 19:34:05.771295 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.771097 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:34:05.771295 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.771132 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:34:05.929619 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.929581 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5hwp" event={"ID":"e8699c88-8f03-4a7b-90f7-152a0118d517","Type":"ContainerStarted","Data":"87b1778ec9335f53bd18263bb48fc7a8821a06f290dcfe5a23616b4d0c5a6525"} Oct 30 19:34:05.932884 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.932845 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="7be4539c7b327565785520a4c98cf5275726db0fff93d09ec4223d8c92fcdca3" exitCode=0 Oct 30 19:34:05.933011 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.932937 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"7be4539c7b327565785520a4c98cf5275726db0fff93d09ec4223d8c92fcdca3"} Oct 30 19:34:05.935354 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.935288 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jw9s8" event={"ID":"7cde5a04-1c75-44b3-8e49-bda52bb78768","Type":"ContainerStarted","Data":"3a62cf2e072f919bf4c3768038fb79ad2f060384f10f03b1aab4587869ed1ab7"} Oct 30 19:34:05.976576 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:05.976458 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-l5hwp" podStartSLOduration=2.6856409169999997 podStartE2EDuration="10.976440468s" totalImagesPullingTime="8.290799551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:05.97633476 +0000 UTC m=+11.679186252" watchObservedRunningTime="2026-10-30 19:34:05.976440468 +0000 UTC m=+11.679291944" Oct 30 19:34:06.017836 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.017770 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jw9s8" podStartSLOduration=1.655936828 podStartE2EDuration="10.017748747s" totalImagesPullingTime="8.361811919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:06.017301506 +0000 UTC m=+11.720152983" watchObservedRunningTime="2026-10-30 19:34:06.017748747 +0000 UTC m=+11.720600222" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.772750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.772763 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.772778 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:06.772869 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:06.773024 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:06.773166 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:06.773097 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:06.941294 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.941040 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" event={"ID":"ea3ab265-3e11-41d3-9192-e40c7588e1f5","Type":"ContainerStarted","Data":"0c8735f59fcc3665bd674f801e5f217ca9426b54b04d5823483a69342d42048c"} Oct 30 19:34:06.983598 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:06.983544 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" podStartSLOduration=3.498262297 podStartE2EDuration="11.983525513s" totalImagesPullingTime="8.485263216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:06.98324079 +0000 UTC m=+12.686092263" watchObservedRunningTime="2026-10-30 19:34:06.983525513 +0000 UTC m=+12.686376985" Oct 30 19:34:07.945611 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:07.945568 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qzmns" event={"ID":"9dcadcb7-1c64-4956-bf5c-489946be6487","Type":"ContainerStarted","Data":"ef209c294114287b43bc17a9f5c64c067b0f707a8d2d13f1494605e40a76c537"} Oct 30 19:34:07.948452 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:07.948419 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-przhx" event={"ID":"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e","Type":"ContainerStarted","Data":"ea318a89af97c656f1c9e0b0f287b6dabccae01d94959b7bf0acaaee47fd77f5"} Oct 30 19:34:07.950721 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:07.950694 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="1ca0891ba6e95bcd534028e75d7036edddabeda66d113b9956da45b828fa1048" exitCode=0 Oct 30 19:34:07.950854 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:07.950789 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"1ca0891ba6e95bcd534028e75d7036edddabeda66d113b9956da45b828fa1048"} Oct 30 19:34:07.987506 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:07.987446 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qzmns" podStartSLOduration=2.265008549 podStartE2EDuration="12.98742702s" totalImagesPullingTime="10.722418471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:07.986948457 +0000 UTC m=+13.689799929" watchObservedRunningTime="2026-10-30 19:34:07.98742702 +0000 UTC m=+13.690278676" Oct 30 19:34:08.070558 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:08.070439 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-przhx" podStartSLOduration=4.818796606 podStartE2EDuration="13.070418772s" totalImagesPullingTime="8.251622166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:08.024900472 +0000 UTC m=+13.727751944" watchObservedRunningTime="2026-10-30 19:34:08.070418772 +0000 UTC m=+13.773270293" Oct 30 19:34:08.773278 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:08.773242 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:08.773460 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:08.773253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:08.773460 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:08.773370 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:08.773460 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:08.773394 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:08.773659 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:08.773548 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:08.773659 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:08.773644 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:09.958302 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:09.958271 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="b7f2cae624f18dec22e835538d8aebcb964833d55bf7d11e17809979f9b4687c" exitCode=0 Oct 30 19:34:09.959127 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:09.958320 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"b7f2cae624f18dec22e835538d8aebcb964833d55bf7d11e17809979f9b4687c"} Oct 30 19:34:10.773468 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:10.773422 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:10.773702 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:10.773487 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:10.773702 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:10.773487 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:10.773702 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:10.773604 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:10.773702 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:10.773664 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:10.773934 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:10.773725 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:11.965453 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:11.965251 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="f6689b11f01b57805e62ab974f892e223a96ab993c4c5da6a5d622303757b4cc" exitCode=0 Oct 30 19:34:11.965802 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:11.965327 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"f6689b11f01b57805e62ab974f892e223a96ab993c4c5da6a5d622303757b4cc"} Oct 30 19:34:12.556406 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.556383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:12.556531 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.556414 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:12.556569 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.556547 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:12.556608 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.556600 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.556587091 +0000 UTC m=+34.259438540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:34:12.556646 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.556547 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:12.556646 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.556635 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret podName:b001525d-fb68-4e3f-b967-f1a8d43f620b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.556629169 +0000 UTC m=+34.259480631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret") pod "global-pull-secret-syncer-shwzj" (UID: "b001525d-fb68-4e3f-b967-f1a8d43f620b") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:34:12.657136 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.657109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:12.657284 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.657266 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:34:12.657284 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.657285 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:34:12.657389 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.657295 2571 projected.go:196] Error preparing data for projected volume kube-api-access-xqwlv for pod openshift-network-diagnostics/network-check-target-8f2fj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:12.657389 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.657347 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv podName:3fc71932-8adb-4e3a-83e8-acfe87c1c896 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.657333186 +0000 UTC m=+34.360184668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqwlv" (UniqueName: "kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv") pod "network-check-target-8f2fj" (UID: "3fc71932-8adb-4e3a-83e8-acfe87c1c896") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:34:12.773448 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.773414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:12.773603 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.773422 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:12.773603 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.773566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:12.773603 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.773568 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:12.773764 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.773649 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:12.773764 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:12.773721 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:12.971016 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.970917 2571 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="3d9c6dd57e2efbc01de05b0a6662772e5fee7414ce76ce4ee056479de7f62a94" exitCode=0 Oct 30 19:34:12.971406 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.971011 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerDied","Data":"3d9c6dd57e2efbc01de05b0a6662772e5fee7414ce76ce4ee056479de7f62a94"} Oct 30 19:34:12.974077 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974055 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"add12e66d43405a9a3d305ba9a1fcf9f24e1fe00744206ebff2297ac8cfc477a"} Oct 30 19:34:12.974129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974121 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"dd1196448abf064570c22d0d0e426d07c212bc9bf4f7c6afee11fd8683828e74"} Oct 30 19:34:12.974158 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974152 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"febed168ef5ed5eaae4d74524f5fda51ea4aadabf3df7a89ac56b722630087c9"} Oct 30 19:34:12.974197 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974187 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"0d7dad43f21e588f37744895012b7b28fcd2cd038729f0a20a9c98ce81c2789e"} Oct 30 19:34:12.974226 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974219 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"8df7cf318fc431a91bda4d631703369d2f498920d51fc3496a20f9b7fa71764c"} Oct 30 19:34:12.974254 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:12.974247 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"c0f0d8d65e70c81ac7d02b2a480685208647e193f9d14d60dbad9cc6e7cb8afa"} Oct 30 19:34:13.979710 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:13.979430 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" event={"ID":"ed54fa06-9be0-44dc-bf64-f48ea87b081f","Type":"ContainerStarted","Data":"b98be03eece7d819c5858be88dd9dd8261cc56dea5f6d3b8efd08747598d3a0e"} Oct 30 19:34:14.050550 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:14.050487 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k7rbb" podStartSLOduration=7.888992833 podStartE2EDuration="19.050473466s" totalImagesPullingTime="11.161480633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:14.03834515 +0000 UTC m=+19.741196622" watchObservedRunningTime="2026-10-30 19:34:14.050473466 +0000 UTC m=+19.753324915" Oct 30 19:34:14.773939 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:14.773904 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:14.774089 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:14.774005 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:14.774089 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:14.774078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:14.774197 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:14.774171 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:14.774240 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:14.774219 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:14.774326 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:14.774299 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:14.985239 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:14.985200 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"dbce245a7c6d2f820652c27c339d2aa3188eb9987f41c9a1d530e8a40a9567aa"} Oct 30 19:34:16.773300 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.773262 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:16.773300 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.773292 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:16.773836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:16.773358 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:16.773836 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.773403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:16.773836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:16.773557 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:16.773836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:16.773636 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:16.994636 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.994602 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" event={"ID":"a6c85830-f826-4df6-a48f-372c870e653a","Type":"ContainerStarted","Data":"9c2bc3dea03ecb5dd0e7494df181cf18098e48f1fbbdffdbac49df090891d881"} Oct 30 19:34:16.995040 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.995013 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:16.995188 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:16.995094 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:17.012074 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:17.012039 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:17.012684 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:17.012661 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:17.088209 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:17.088090 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" podStartSLOduration=6.738589316 podStartE2EDuration="22.088072875s" totalImagesPullingTime="15.349483559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:17.075482322 +0000 UTC m=+22.778333794" watchObservedRunningTime="2026-10-30 19:34:17.088072875 +0000 UTC m=+22.790924347" Oct 30 19:34:17.997374 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:17.997344 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:18.773091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:18.773057 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:18.773091 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:18.773077 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:18.773344 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:18.773083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:18.773344 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:18.773171 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:18.773344 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:18.773241 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:18.773344 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:18.773306 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:19.000250 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.000227 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:19.086743 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.086629 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8f2fj"] Oct 30 19:34:19.086918 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.086759 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:19.086918 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:19.086863 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:19.089607 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.089550 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57wdt"] Oct 30 19:34:19.089742 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.089685 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:19.089825 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:19.089803 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:19.099046 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.099008 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shwzj"] Oct 30 19:34:19.099204 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:19.099109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:19.099204 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:19.099193 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:20.773324 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:20.773151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:20.773805 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:20.773151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:20.773805 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:20.773425 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:20.773805 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:20.773155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:20.773805 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:20.773476 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:20.773805 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:20.773583 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:22.773195 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:22.773146 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:22.773832 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:22.773164 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:22.773832 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:22.773300 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:22.773832 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:22.773146 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:22.773832 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:22.773370 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:22.773832 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:22.773443 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:24.774110 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:24.774082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:24.774477 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:24.774194 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:24.774477 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:24.774213 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57wdt" podUID="36088f70-cd5c-4911-ac09-cfb10529c20a" Oct 30 19:34:24.774477 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:24.774236 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:24.774477 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:24.774307 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shwzj" podUID="b001525d-fb68-4e3f-b967-f1a8d43f620b" Oct 30 19:34:24.774477 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:24.774358 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8f2fj" podUID="3fc71932-8adb-4e3a-83e8-acfe87c1c896" Oct 30 19:34:25.153420 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.153392 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeReady" Oct 30 19:34:25.153586 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.153554 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:34:25.206089 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.206054 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kv78f"] Oct 30 19:34:25.209259 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.209241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:25.210697 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.210671 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h4vsd"] Oct 30 19:34:25.213614 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.213595 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7t7rs"] Oct 30 19:34:25.213716 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.213698 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:25.216272 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.216249 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:34:25.216365 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.216346 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:25.219016 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.218926 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:34:25.219088 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.219042 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.221703 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.221683 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-frzlq"] Oct 30 19:34:25.221798 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.221779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.225425 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.225404 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mb2fw"] Oct 30 19:34:25.225601 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.225581 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.228257 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.228240 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp"] Oct 30 19:34:25.228371 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.228356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.231678 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.231646 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zl8sn"] Oct 30 19:34:25.231769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.231732 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.231769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.231744 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.231878 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.231819 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.234442 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.234427 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c"] Oct 30 19:34:25.234572 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.234562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.237017 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.236999 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz"] Oct 30 19:34:25.237137 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.237118 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.239892 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.239869 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5"] Oct 30 19:34:25.240485 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.240465 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:25.250957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.250935 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-fd9894b6f-skqbz"] Oct 30 19:34:25.251207 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.251185 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.253644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.253629 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5484c"] Oct 30 19:34:25.253763 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.253748 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.256661 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.256482 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl"] Oct 30 19:34:25.256661 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.256590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.262081 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262059 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sg9nh"] Oct 30 19:34:25.262253 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262149 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262771 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lkchj\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262809 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262876 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.262907 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2hnnk\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.263223 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.263285 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.263318 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:34:25.263729 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.263383 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:34:25.266690 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.266666 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r65qm"] Oct 30 19:34:25.271547 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.267227 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.271547 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.267294 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9nnsl\"" Oct 30 19:34:25.271905 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271883 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kv78f"] Oct 30 19:34:25.271957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271915 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h4vsd"] Oct 30 19:34:25.271957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271935 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7t7rs"] Oct 30 19:34:25.271957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271956 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-frzlq"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271974 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271992 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.271994 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.272012 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.272038 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zl8sn"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.272064 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5484c"] Oct 30 19:34:25.272097 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.272083 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sg9nh"] Oct 30 19:34:25.272358 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.272107 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fd9894b6f-skqbz"] Oct 30 19:34:25.278234 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.278211 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c"] Oct 30 19:34:25.281082 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.281057 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:34:25.294421 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.294399 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mb2fw"] Oct 30 19:34:25.301836 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.301651 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r65qm"] Oct 30 19:34:25.301836 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.301700 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz"] Oct 30 19:34:25.310485 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.310464 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl"] Oct 30 19:34:25.340298 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.340274 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.341280 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341258 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.341670 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341633 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.341804 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341656 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.341898 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341874 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.341898 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341870 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:34:25.342053 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.341750 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.342131 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.342108 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:34:25.343577 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.343558 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.343685 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.343607 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.343799 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.343787 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.344173 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.344147 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.344269 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.344251 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.344725 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.344706 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:34:25.345169 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.344743 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.345621 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.345601 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:34:25.345788 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.345769 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:34:25.346013 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.345992 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.346107 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346030 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.346300 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf79p\" (UniqueName: \"kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.346358 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346313 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.346358 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk5m8\" (UniqueName: \"kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8\") pod \"network-check-source-6f78d59f99-l2ccl\" (UID: \"46fd767b-686b-42dd-acdd-5901294699ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:25.346450 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346359 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.346450 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.346450 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.346618 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346463 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.346618 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.346618 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346544 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr87n\" (UniqueName: \"kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.346618 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zghxs\" (UniqueName: \"kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346710 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvtz\" (UniqueName: \"kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346741 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbvg\" (UniqueName: \"kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346780 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.346834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346813 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.347182 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.347182 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.346972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.347182 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.347182 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.347182 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.347400 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347194 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.347400 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.347400 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.347400 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347321 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.347400 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347363 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk4s5\" (UniqueName: \"kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5\") pod \"kas-connection-checker-685895c7c9-kv78f\" (UID: \"df601210-c5b1-4f3f-939b-bb8a11c3759d\") " pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347464 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.347644 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tsd\" (UniqueName: \"kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347706 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347734 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ql9v\" (UniqueName: \"kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v\") pod \"kas-connection-checker-685895c7c9-h4vsd\" (UID: \"fc0ef185-0820-4ae9-955b-0f35510667d9\") " pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7ps\" (UniqueName: \"kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps\") pod \"volume-data-source-validator-58c9f5f46d-ppsxz\" (UID: \"3da0ba8c-d364-4754-9069-47fe4a2017a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:25.347888 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347841 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxfk\" (UniqueName: \"kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk\") pod \"kas-connection-checker-685895c7c9-7t7rs\" (UID: \"16eeabd2-8f23-411a-b951-6207f95bdb43\") " pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:25.348154 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.348154 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.347980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.348154 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348025 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.348154 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx5r\" (UniqueName: \"kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.348332 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348222 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.348332 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.348418 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348347 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.348465 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348411 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.348465 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.348675 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348581 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.348804 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.348880 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348856 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfbf\" (UniqueName: \"kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.348937 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhm7\" (UniqueName: \"kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.348991 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.348977 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.349096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.349071 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.349140 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.349128 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.349200 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.349175 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.349586 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.349565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.351472 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351452 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:34:25.351570 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351452 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.351570 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351457 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.351570 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351558 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:34:25.351706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351599 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:25.351706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351608 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:34:25.351706 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351678 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.351831 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.351753 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:25.355210 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.355188 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:34:25.355522 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.355501 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:34:25.355825 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.355700 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:34:25.356010 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.355987 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364025 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364045 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364047 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364056 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l9ldq\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364047 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:34:25.364096 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364053 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364113 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364287 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364293 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364305 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364321 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9pzwv\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364326 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364333 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364351 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sqtpr\"" Oct 30 19:34:25.364440 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364432 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m8v7z\"" Oct 30 19:34:25.364726 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364463 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fkztb\"" Oct 30 19:34:25.364726 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.364469 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p87n7\"" Oct 30 19:34:25.424472 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.424388 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:25.450670 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.450670 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450672 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.450879 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450695 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.450879 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.450879 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.450788 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:25.450879 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.450848 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.950828308 +0000 UTC m=+31.653679771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:25.450879 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfbf\" (UniqueName: \"kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.451129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450896 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhm7\" (UniqueName: \"kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.451129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450918 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.451129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.450943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.451129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.451129 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451156 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf79p\" (UniqueName: \"kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5m8\" (UniqueName: \"kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8\") pod \"network-check-source-6f78d59f99-l2ccl\" (UID: \"46fd767b-686b-42dd-acdd-5901294699ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451259 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.451355 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr87n\" (UniqueName: \"kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghxs\" (UniqueName: \"kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451484 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451516 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvtz\" (UniqueName: \"kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451665 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbvg\" (UniqueName: \"kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.451766 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451885 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451929 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452001 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452081 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452123 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4s5\" (UniqueName: \"kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5\") pod \"kas-connection-checker-685895c7c9-kv78f\" (UID: \"df601210-c5b1-4f3f-939b-bb8a11c3759d\") " pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.452327 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tsd\" (UniqueName: \"kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452544 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ql9v\" (UniqueName: \"kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v\") pod \"kas-connection-checker-685895c7c9-h4vsd\" (UID: \"fc0ef185-0820-4ae9-955b-0f35510667d9\") " pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452677 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7ps\" (UniqueName: \"kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps\") pod \"volume-data-source-validator-58c9f5f46d-ppsxz\" (UID: \"3da0ba8c-d364-4754-9069-47fe4a2017a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxfk\" (UniqueName: \"kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk\") pod \"kas-connection-checker-685895c7c9-7t7rs\" (UID: \"16eeabd2-8f23-411a-b951-6207f95bdb43\") " pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452705 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx5r\" (UniqueName: \"kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.453220 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.453120 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.453977 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.451858 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.453977 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.453646 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.453977 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.453882 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.454861 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.454902 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.451955 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.455357 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.455437 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.455441 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.455452 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.455458 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.452677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.455728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.455824 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.456013 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.456142 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.453008 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:25.456867 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.454592 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:25.456867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.456373 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.456867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.456614 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.456867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.456669 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.457205 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.457187 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.457416 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.457392 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.457819 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457799 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.957776883 +0000 UTC m=+31.660628336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:25.457913 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457828 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.957818889 +0000 UTC m=+31.660670338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:25.457913 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457842 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.957835328 +0000 UTC m=+31.660686783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:25.457913 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457857 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.95784818 +0000 UTC m=+31.660699631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:25.457913 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457872 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.957865501 +0000 UTC m=+31.660716950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:25.457913 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.457890 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.957881031 +0000 UTC m=+31.660732487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:25.458189 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.457937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.458239 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.458209 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:25.458286 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.458257 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.458286 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.458263 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:25.958248583 +0000 UTC m=+31.661100033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:25.458738 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.458715 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.458817 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.458740 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.458933 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.458917 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.459112 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.459091 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.459834 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.459817 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.459979 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.459961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.460059 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.460040 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.475591 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.475569 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhm7\" (UniqueName: \"kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.477122 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.476696 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvtz\" (UniqueName: \"kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.477122 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.476738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfbf\" (UniqueName: \"kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.477307 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.477284 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf79p\" (UniqueName: \"kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.478669 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.478645 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4s5\" (UniqueName: \"kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5\") pod \"kas-connection-checker-685895c7c9-kv78f\" (UID: \"df601210-c5b1-4f3f-939b-bb8a11c3759d\") " pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:25.479544 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.478926 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tsd\" (UniqueName: \"kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.479544 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.479462 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.480037 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.480002 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5m8\" (UniqueName: \"kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8\") pod \"network-check-source-6f78d59f99-l2ccl\" (UID: \"46fd767b-686b-42dd-acdd-5901294699ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:25.480751 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.480728 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.481313 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.481288 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7ps\" (UniqueName: \"kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps\") pod \"volume-data-source-validator-58c9f5f46d-ppsxz\" (UID: \"3da0ba8c-d364-4754-9069-47fe4a2017a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:25.482549 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482086 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghxs\" (UniqueName: \"kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.482549 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482105 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ql9v\" (UniqueName: \"kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v\") pod \"kas-connection-checker-685895c7c9-h4vsd\" (UID: \"fc0ef185-0820-4ae9-955b-0f35510667d9\") " pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:25.482549 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482397 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxfk\" (UniqueName: \"kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk\") pod \"kas-connection-checker-685895c7c9-7t7rs\" (UID: \"16eeabd2-8f23-411a-b951-6207f95bdb43\") " pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:25.482549 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482515 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbvg\" (UniqueName: \"kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.482798 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482785 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.482844 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482797 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.482950 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.482932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx5r\" (UniqueName: \"kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.483631 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.483607 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr87n\" (UniqueName: \"kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.485555 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.485485 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:25.485812 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.485787 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:25.497419 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.497393 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:25.525085 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.525063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:25.531288 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.531272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:25.538448 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.538413 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:25.563471 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.563436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:25.590522 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.590446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:25.601075 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.598670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:25.610877 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.610679 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:25.613586 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.613476 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:25.633539 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.632672 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:25.633539 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.633417 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:25.650183 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.650030 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:25.850575 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.850554 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h4vsd"] Oct 30 19:34:25.852277 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:25.852235 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0ef185_0820_4ae9_955b_0f35510667d9.slice/crio-d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b WatchSource:0}: Error finding container d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b: Status 404 returned error can't find the container with id d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b Oct 30 19:34:25.855980 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.854650 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7t7rs"] Oct 30 19:34:25.856098 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.856077 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kv78f"] Oct 30 19:34:25.856302 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:25.856256 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eeabd2_8f23_411a_b951_6207f95bdb43.slice/crio-79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b WatchSource:0}: Error finding container 79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b: Status 404 returned error can't find the container with id 79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b Oct 30 19:34:25.857295 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:25.857254 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf601210_c5b1_4f3f_939b_bb8a11c3759d.slice/crio-587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e WatchSource:0}: Error finding container 587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e: Status 404 returned error can't find the container with id 587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e Oct 30 19:34:25.912136 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.912104 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-frzlq"] Oct 30 19:34:25.918318 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:25.918262 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fa447b_1c5e_4eab_aeda_1c0f6d80f1ae.slice/crio-5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7 WatchSource:0}: Error finding container 5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7: Status 404 returned error can't find the container with id 5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7 Oct 30 19:34:25.963005 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.962982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:25.963089 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:25.963089 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.963089 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:25.963228 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:25.963228 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:25.963228 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963216 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.963271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963289 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963272159 +0000 UTC m=+32.666123609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963310 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963321 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963304801 +0000 UTC m=+32.666156279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963323 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963338 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963356 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963347262 +0000 UTC m=+32.666198710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:25.963371 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963222 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963382 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963368322 +0000 UTC m=+32.666219786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963222 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963399 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963390169 +0000 UTC m=+32.666241621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963260 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963260 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963421 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963415 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963406712 +0000 UTC m=+32.666258167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963455 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963448116 +0000 UTC m=+32.666299566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:25.963710 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:25.963465 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:26.963460282 +0000 UTC m=+32.666311731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:25.968890 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.968866 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz"] Oct 30 19:34:25.971467 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.971409 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zl8sn"] Oct 30 19:34:25.974886 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.974837 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c"] Oct 30 19:34:25.982898 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:25.982654 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5"] Oct 30 19:34:25.993907 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:25.993871 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f9c455_a5a6_4c80_8bcd_cd0fe9d4d374.slice/crio-f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51 WatchSource:0}: Error finding container f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51: Status 404 returned error can't find the container with id f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51 Oct 30 19:34:26.017251 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.017187 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" event={"ID":"410e9153-45cc-485e-ba92-efa60cca301e","Type":"ContainerStarted","Data":"7d6ab31f712825ba8a00059f7273d53808e2eeebaa3bac54bd9f9dc186438e62"} Oct 30 19:34:26.018634 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.018606 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-frzlq" event={"ID":"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae","Type":"ContainerStarted","Data":"5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7"} Oct 30 19:34:26.020029 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.020008 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" event={"ID":"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374","Type":"ContainerStarted","Data":"f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51"} Oct 30 19:34:26.028478 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.028448 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl"] Oct 30 19:34:26.030353 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.030328 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" event={"ID":"3da0ba8c-d364-4754-9069-47fe4a2017a6","Type":"ContainerStarted","Data":"c7232937f529ecdfecf34679747e36d3f76678ca6fe46f323caefb97aeb84048"} Oct 30 19:34:26.031516 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.031476 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" event={"ID":"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4","Type":"ContainerStarted","Data":"97e56f8114c184e3b1077fc75f28f433cdeaeb916fefe395e19864ce66611028"} Oct 30 19:34:26.032909 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.032891 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kv78f" event={"ID":"df601210-c5b1-4f3f-939b-bb8a11c3759d","Type":"ContainerStarted","Data":"62c53ef3e43205387d7c34c9f06ab509608de1ed596e882041da68a38be55c87"} Oct 30 19:34:26.032985 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.032921 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kv78f" event={"ID":"df601210-c5b1-4f3f-939b-bb8a11c3759d","Type":"ContainerStarted","Data":"587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e"} Oct 30 19:34:26.034197 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:26.034173 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fd767b_686b_42dd_acdd_5901294699ac.slice/crio-30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8 WatchSource:0}: Error finding container 30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8: Status 404 returned error can't find the container with id 30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8 Oct 30 19:34:26.034384 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.034361 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" event={"ID":"16eeabd2-8f23-411a-b951-6207f95bdb43","Type":"ContainerStarted","Data":"5ee1f12af8f9523872db48aae5e912b64b62d54412700efdec7c4050fa0bc0f6"} Oct 30 19:34:26.034446 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.034401 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" event={"ID":"16eeabd2-8f23-411a-b951-6207f95bdb43","Type":"ContainerStarted","Data":"79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b"} Oct 30 19:34:26.035717 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.035694 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" event={"ID":"fc0ef185-0820-4ae9-955b-0f35510667d9","Type":"ContainerStarted","Data":"17c0d2e0f5fd0a552f8c4457722d291ec42b0b060c12ddfb2560e2814174726f"} Oct 30 19:34:26.035814 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.035734 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" event={"ID":"fc0ef185-0820-4ae9-955b-0f35510667d9","Type":"ContainerStarted","Data":"d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b"} Oct 30 19:34:26.071637 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.071579 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kv78f" podStartSLOduration=403.071564088 podStartE2EDuration="6m43.071564088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:26.070983696 +0000 UTC m=+31.773835169" watchObservedRunningTime="2026-10-30 19:34:26.071564088 +0000 UTC m=+31.774415560" Oct 30 19:34:26.075050 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.075030 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r65qm"] Oct 30 19:34:26.076368 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:26.076344 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1ef3c9_11c5_48c2_b1db_166270b55b09.slice/crio-e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d WatchSource:0}: Error finding container e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d: Status 404 returned error can't find the container with id e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d Oct 30 19:34:26.111059 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.111015 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" podStartSLOduration=403.111003542 podStartE2EDuration="6m43.111003542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:26.109721993 +0000 UTC m=+31.812573463" watchObservedRunningTime="2026-10-30 19:34:26.111003542 +0000 UTC m=+31.813855013" Oct 30 19:34:26.153531 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.153462 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" podStartSLOduration=403.153446691 podStartE2EDuration="6m43.153446691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:26.152578962 +0000 UTC m=+31.855430435" watchObservedRunningTime="2026-10-30 19:34:26.153446691 +0000 UTC m=+31.856298163" Oct 30 19:34:26.334807 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.334735 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.440442 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.440403 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:26.780921 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.780880 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:26.781986 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.781375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:26.781986 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.781603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:26.811141 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.811118 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:34:26.811247 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.811159 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:34:26.811247 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.811186 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\"" Oct 30 19:34:26.811247 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.811240 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\"" Oct 30 19:34:26.811422 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.811272 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\"" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975126 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975225 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:26.975376 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975535 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975599 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.975579787 +0000 UTC m=+34.678431253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975698 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975710 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975746 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.975733629 +0000 UTC m=+34.678585092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975804 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975813 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.975840 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.975831052 +0000 UTC m=+34.678682516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:26.976529 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976210 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.976238606 +0000 UTC m=+34.679090072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976312 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976344 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.976333782 +0000 UTC m=+34.679185246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976402 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976430 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.976419753 +0000 UTC m=+34.679271219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:26.977836 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976485 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:26.978301 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.976652 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.976632495 +0000 UTC m=+34.679483984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:26.978301 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:26.978271 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:28.978251887 +0000 UTC m=+34.681103339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:27.046661 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:27.046537 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r65qm" event={"ID":"ae1ef3c9-11c5-48c2-b1db-166270b55b09","Type":"ContainerStarted","Data":"e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d"} Oct 30 19:34:27.050904 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:27.050861 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" event={"ID":"46fd767b-686b-42dd-acdd-5901294699ac","Type":"ContainerStarted","Data":"30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8"} Oct 30 19:34:28.597746 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.596982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:28.597746 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.597097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:28.597746 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:28.597732 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:28.598274 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:28.597796 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:35:00.597777424 +0000 UTC m=+66.300628874 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:28.602483 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.602109 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:28.698955 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.698721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:28.710914 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.710876 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:28.898904 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.898673 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:28.949441 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:28.949413 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:29.002609 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:29.002738 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002668 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:29.002738 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002702 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:29.002738 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:29.002895 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002780 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:29.002895 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:29.002895 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:29.002895 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.002890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:29.004042 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003046 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.003025423 +0000 UTC m=+38.705876895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005549 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005619 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005605259 +0000 UTC m=+38.708456727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003228 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005678 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005707 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005698661 +0000 UTC m=+38.708550110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003170 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005743 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005737567 +0000 UTC m=+38.708589016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003204 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005763 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005759037 +0000 UTC m=+38.708610485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003209 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005773 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005789 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005784354 +0000 UTC m=+38.708635803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003259 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005812 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005807451 +0000 UTC m=+38.708658900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.003265 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:29.005911 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:29.005834 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:33.005828125 +0000 UTC m=+38.708679574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:29.092406 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.091872 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" event={"ID":"3da0ba8c-d364-4754-9069-47fe4a2017a6","Type":"ContainerStarted","Data":"af41a77463c73e35987cfc6ccf23a69fef6cf1635c8d5a606585620e370ee174"} Oct 30 19:34:29.137321 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.137263 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" podStartSLOduration=361.526476739 podStartE2EDuration="6m4.137250795s" totalImagesPullingTime="2.610774056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:29.136290765 +0000 UTC m=+34.839142237" watchObservedRunningTime="2026-10-30 19:34:29.137250795 +0000 UTC m=+34.840102263" Oct 30 19:34:29.255637 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.255460 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shwzj"] Oct 30 19:34:29.257916 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.257867 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:29.260238 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:29.260201 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb001525d_fb68_4e3f_b967_f1a8d43f620b.slice/crio-ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85 WatchSource:0}: Error finding container ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85: Status 404 returned error can't find the container with id ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85 Oct 30 19:34:29.313023 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.313000 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8f2fj"] Oct 30 19:34:29.315206 ip-10-0-132-115 kubenswrapper[2571]: W1030 19:34:29.315174 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc71932_8adb_4e3a_83e8_acfe87c1c896.slice/crio-0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d WatchSource:0}: Error finding container 0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d: Status 404 returned error can't find the container with id 0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d Oct 30 19:34:29.366968 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:29.366880 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:30.100569 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.100285 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-frzlq" event={"ID":"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae","Type":"ContainerStarted","Data":"ba16583277c4ab7778acb70aead594f5d71b9f8575720c2c649948ebf92bbfc5"} Oct 30 19:34:30.103935 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.103907 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shwzj" event={"ID":"b001525d-fb68-4e3f-b967-f1a8d43f620b","Type":"ContainerStarted","Data":"ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85"} Oct 30 19:34:30.106268 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.106245 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8f2fj" event={"ID":"3fc71932-8adb-4e3a-83e8-acfe87c1c896","Type":"ContainerStarted","Data":"0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d"} Oct 30 19:34:30.143786 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.143450 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-frzlq" podStartSLOduration=368.265395012 podStartE2EDuration="6m12.143431621s" totalImagesPullingTime="3.878036609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:30.141602262 +0000 UTC m=+35.844453733" watchObservedRunningTime="2026-10-30 19:34:30.143431621 +0000 UTC m=+35.846283094" Oct 30 19:34:30.829118 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.827910 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q8jlb"] Oct 30 19:34:30.848642 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.848612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:30.853867 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.853823 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8jlb"] Oct 30 19:34:30.864559 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.864533 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:34:30.864777 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.864590 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:34:30.864777 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.864704 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\"" Oct 30 19:34:30.895049 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:30.895000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:31.006152 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.006125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:31.031756 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.031456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2dh\" (UniqueName: \"kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.032769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.032407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.032769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.032558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.032769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.032592 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.032769 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.032617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.112269 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.112172 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" event={"ID":"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374","Type":"ContainerStarted","Data":"44febf9f6dc5a8b18f587d55d7c49b800f8a58abe3ce7f3849948176c85eac43"} Oct 30 19:34:31.113973 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.113951 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/0.log" Oct 30 19:34:31.114080 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.113989 2571 generic.go:356] "Generic (PLEG): container finished" podID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerID="69a53404e18b31f456e576c9813434ddd8cb8c42edca173ce9b582c8deb68af7" exitCode=255 Oct 30 19:34:31.114448 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.114424 2571 scope.go:122] "RemoveContainer" containerID="69a53404e18b31f456e576c9813434ddd8cb8c42edca173ce9b582c8deb68af7" Oct 30 19:34:31.114577 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.114516 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" event={"ID":"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4","Type":"ContainerDied","Data":"69a53404e18b31f456e576c9813434ddd8cb8c42edca173ce9b582c8deb68af7"} Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.133614 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.133653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.133774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2dh\" (UniqueName: \"kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.133821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.133968 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.134308 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.134382 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.134438 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.634418558 +0000 UTC m=+37.337270012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : configmap "kube-rbac-proxy" not found Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.134530 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.134565 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:31.634554157 +0000 UTC m=+37.337405609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:31.135087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.134829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.152957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.152192 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" podStartSLOduration=361.205042886 podStartE2EDuration="6m6.152181208s" totalImagesPullingTime="4.947138322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:31.151488436 +0000 UTC m=+36.854339910" watchObservedRunningTime="2026-10-30 19:34:31.152181208 +0000 UTC m=+36.855032679" Oct 30 19:34:31.169219 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.169192 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2dh\" (UniqueName: \"kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.640382 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.640353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.640483 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.640397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:31.640695 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.640624 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:31.640695 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.640690 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:32.6406725 +0000 UTC m=+38.343523964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:31.640811 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.640736 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:34:31.640811 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:31.640769 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:32.640758214 +0000 UTC m=+38.343609677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : configmap "kube-rbac-proxy" not found Oct 30 19:34:31.995087 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:31.994995 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.049015 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.048983 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk"] Oct 30 19:34:32.066209 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.066183 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:32.075022 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.074919 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk"] Oct 30 19:34:32.105638 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.105605 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-khwrx\"" Oct 30 19:34:32.124002 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.123827 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:32.124002 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.123949 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:34:32.125681 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.125612 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:32.126940 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.126159 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/0.log" Oct 30 19:34:32.126940 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.126191 2571 generic.go:356] "Generic (PLEG): container finished" podID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" exitCode=255 Oct 30 19:34:32.126940 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.126272 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" event={"ID":"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4","Type":"ContainerDied","Data":"1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd"} Oct 30 19:34:32.126940 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.126319 2571 scope.go:122] "RemoveContainer" containerID="69a53404e18b31f456e576c9813434ddd8cb8c42edca173ce9b582c8deb68af7" Oct 30 19:34:32.126940 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.126758 2571 scope.go:122] "RemoveContainer" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" Oct 30 19:34:32.127928 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:32.127352 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:32.131754 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.131243 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r65qm" event={"ID":"ae1ef3c9-11c5-48c2-b1db-166270b55b09","Type":"ContainerStarted","Data":"67df18c24dea65143ddb0240437a407d3e2965e33bc556d4cc30e45d360cc7a7"} Oct 30 19:34:32.136217 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.136168 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" event={"ID":"410e9153-45cc-485e-ba92-efa60cca301e","Type":"ContainerStarted","Data":"d685ed919c5d7bba51c1c66f2608858a9c1fe6b4e050a4e876bded70a75a37c4"} Oct 30 19:34:32.160957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.160839 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.192750 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.192695 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" podStartSLOduration=361.631064624 podStartE2EDuration="6m7.192673804s" totalImagesPullingTime="5.56160918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:32.191042834 +0000 UTC m=+37.893894309" watchObservedRunningTime="2026-10-30 19:34:32.192673804 +0000 UTC m=+37.895525277" Oct 30 19:34:32.247750 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.247266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s4x\" (UniqueName: \"kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x\") pod \"migrator-66df5d496b-sddsk\" (UID: \"34d9f428-c41a-47ae-9ae5-9d93c76941ef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:32.278793 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.278648 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.288627 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.287154 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r65qm" podStartSLOduration=1.453670295 podStartE2EDuration="7.287139204s" totalImagesPullingTime="5.833468909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:32.237900592 +0000 UTC m=+37.940752075" watchObservedRunningTime="2026-10-30 19:34:32.287139204 +0000 UTC m=+37.989990676" Oct 30 19:34:32.349226 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.349187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s4x\" (UniqueName: \"kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x\") pod \"migrator-66df5d496b-sddsk\" (UID: \"34d9f428-c41a-47ae-9ae5-9d93c76941ef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:32.372107 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.372078 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s4x\" (UniqueName: \"kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x\") pod \"migrator-66df5d496b-sddsk\" (UID: \"34d9f428-c41a-47ae-9ae5-9d93c76941ef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:32.399957 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.399928 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:32.532213 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.532183 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.648300 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.648268 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:32.651903 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.651879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:32.652052 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.651914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:32.652208 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:32.652188 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:34:32.652323 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:32.652256 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:34.652239091 +0000 UTC m=+40.355090544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : configmap "kube-rbac-proxy" not found Oct 30 19:34:32.652323 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:32.652262 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:32.652439 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:32.652326 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:34.652308006 +0000 UTC m=+40.355159457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:32.819363 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:32.819079 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk"] Oct 30 19:34:33.056145 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:33.056145 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056118 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056192 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056212 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056251 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056278 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056260627 +0000 UTC m=+46.759112083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056302 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056289855 +0000 UTC m=+46.759141340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:33.056402 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056371 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056346 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056377 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056465 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056413 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056444 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056437257 +0000 UTC m=+46.759288748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056536 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.05652643 +0000 UTC m=+46.759377882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056548 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056542714 +0000 UTC m=+46.759394163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056559 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056553945 +0000 UTC m=+46.759405394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056581 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056567268 +0000 UTC m=+46.759418717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.056642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056742 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056752 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:33.056892 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.056788 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:41.056778783 +0000 UTC m=+46.759630247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:33.142379 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.142354 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:33.142873 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.142742 2571 scope.go:122] "RemoveContainer" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" Oct 30 19:34:33.142966 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:33.142943 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:33.144485 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.144460 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8f2fj" event={"ID":"3fc71932-8adb-4e3a-83e8-acfe87c1c896","Type":"ContainerStarted","Data":"6a93433b7ed6454b09ce753a2c64c4639c5cb1476803bd7348290cb9b97d16ec"} Oct 30 19:34:33.144609 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.144540 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:33.146398 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.146328 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" event={"ID":"46fd767b-686b-42dd-acdd-5901294699ac","Type":"ContainerStarted","Data":"cdb057b7214dc7996f787740bb63c6d5573602cdf7af07dd3b97b8c4ffcae05d"} Oct 30 19:34:33.148821 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.148798 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" event={"ID":"34d9f428-c41a-47ae-9ae5-9d93c76941ef","Type":"ContainerStarted","Data":"461942b33fa15a28c691eef8a5581c2881aa027ac0df8b987f3e6031407715f4"} Oct 30 19:34:33.229691 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.229635 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" podStartSLOduration=360.760479192 podStartE2EDuration="6m7.229616119s" totalImagesPullingTime="6.469136927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:33.227512038 +0000 UTC m=+38.930363499" watchObservedRunningTime="2026-10-30 19:34:33.229616119 +0000 UTC m=+38.932467589" Oct 30 19:34:33.269246 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.269180 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8f2fj" podStartSLOduration=34.819377437 podStartE2EDuration="38.269162437s" totalImagesPullingTime="3.449785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:33.26874428 +0000 UTC m=+38.971595751" watchObservedRunningTime="2026-10-30 19:34:33.269162437 +0000 UTC m=+38.972013911" Oct 30 19:34:33.356900 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.356734 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:33.482520 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:33.482468 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.026307 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:34.026270 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.527663 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:34.527617 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.632709 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:34.632686 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.670933 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:34.670900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:34.671101 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:34.670937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:34.671101 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:34.671031 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:34:34.671101 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:34.671061 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:34.671101 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:34.671100 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.671085115 +0000 UTC m=+44.373936566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : configmap "kube-rbac-proxy" not found Oct 30 19:34:34.671307 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:34.671116 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.671109935 +0000 UTC m=+44.373961384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:35.158305 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:35.158276 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" event={"ID":"34d9f428-c41a-47ae-9ae5-9d93c76941ef","Type":"ContainerStarted","Data":"bdd957a317ce0ec03113ea338774c1068857810617de2a01bb59b4470087ba30"} Oct 30 19:34:35.197100 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:35.197060 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" podStartSLOduration=1.72643417 podStartE2EDuration="3.197047676s" totalImagesPullingTime="1.470613506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:35.196792669 +0000 UTC m=+40.899644139" watchObservedRunningTime="2026-10-30 19:34:35.197047676 +0000 UTC m=+40.899899144" Oct 30 19:34:35.592774 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:35.592754 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:35.593140 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:35.592794 2571 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:35.593199 ip-10-0-132-115 kubenswrapper[2571]: I1030 19:34:35.593161 2571 scope.go:122] "RemoveContainer" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" Oct 30 19:34:35.593390 ip-10-0-132-115 kubenswrapper[2571]: E1030 19:34:35.593365 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:35.814325 ip-10-0-132-115 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:34:35.827925 ip-10-0-132-115 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:34:35.828135 ip-10-0-132-115 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:34:35.828195 ip-10-0-132-115 systemd[1]: kubelet.service: Consumed 3.107s CPU time. Oct 30 19:34:35.839261 ip-10-0-132-115 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:34:35.905313 ip-10-0-132-115 kubenswrapper[4959]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:35.905313 ip-10-0-132-115 kubenswrapper[4959]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:34:35.905313 ip-10-0-132-115 kubenswrapper[4959]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:35.905313 ip-10-0-132-115 kubenswrapper[4959]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:34:35.907410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907329 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.907410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907411 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907415 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907418 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907421 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907424 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907427 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907435 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907439 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907441 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907444 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907447 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907450 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907452 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907455 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907458 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907460 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907464 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907469 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907473 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.907488 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907476 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907479 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907481 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907484 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907486 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907504 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907508 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907510 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907513 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907516 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907518 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907521 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907523 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907526 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907529 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907531 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907534 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907536 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907539 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.907984 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907542 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907545 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907548 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907550 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907552 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907555 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907557 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907560 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907562 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907565 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907567 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907570 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907573 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907575 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907578 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907581 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907583 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907586 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907589 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.908448 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907591 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907594 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907596 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907599 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907601 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907604 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907606 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907609 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907611 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907615 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907618 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907627 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907632 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907635 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907637 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907641 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907643 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907646 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907648 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.908980 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907655 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907657 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907660 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907662 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907665 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907668 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907670 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907673 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907676 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907679 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907682 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907684 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907687 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907689 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907692 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907694 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907696 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907705 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907708 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907710 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.909437 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907713 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907715 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907717 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907720 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907723 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907725 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907728 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907731 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.907734 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908550 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908559 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908562 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908568 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908570 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908573 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908576 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908578 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908581 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908584 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908586 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.909945 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908589 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908592 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908594 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908597 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908600 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908602 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908604 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908607 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908610 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908612 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908615 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908617 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908619 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908622 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908625 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908627 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908629 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908632 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908634 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908637 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.910438 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908640 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908642 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908645 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908648 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908653 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908656 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908658 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908661 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908663 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908666 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908668 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908671 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908673 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908676 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908678 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908681 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908683 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908689 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908691 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.911049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908694 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908698 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908702 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908705 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908708 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908710 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908713 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908715 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908717 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908720 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908722 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908726 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908729 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908732 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908734 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908736 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908744 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908749 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908751 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908754 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.911536 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908757 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908760 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908762 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908764 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908768 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908770 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908773 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908776 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908778 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908781 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908783 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908786 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908788 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908791 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908793 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908796 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908798 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908801 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908803 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.912037 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908806 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908808 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908810 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908813 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908815 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908818 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908820 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908823 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908825 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908827 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908831 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908837 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908840 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908842 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908845 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908847 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.908849 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909403 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909408 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.912514 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909411 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909414 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909417 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909419 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909421 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909424 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909427 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909429 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909431 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909434 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909436 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909439 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909442 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909444 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909447 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909450 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909452 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909455 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909458 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.912998 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909460 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909463 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909465 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909468 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909477 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909480 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909482 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909485 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909487 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909504 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909507 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909510 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909513 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909515 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909518 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909520 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909523 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909525 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909527 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909530 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.913470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909533 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909535 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909537 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909540 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909542 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909545 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909549 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909553 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909556 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909559 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909562 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909565 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909567 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909570 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909572 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909574 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909578 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909586 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909589 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.913988 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909592 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909594 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909597 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909599 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909602 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909606 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909610 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909612 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909616 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909618 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909621 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909623 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909625 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909628 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909630 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909633 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909635 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909638 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909651 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.914728 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909654 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909656 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909659 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909661 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909664 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909666 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909669 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909671 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909674 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909676 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909680 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909682 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909690 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909693 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909696 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909698 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909702 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909704 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909707 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909714 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.915595 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909717 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909719 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909722 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909724 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909726 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909729 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.909731 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909836 4959 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909847 4959 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909855 4959 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909860 4959 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909864 4959 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909868 4959 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909872 4959 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909876 4959 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909879 4959 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909882 4959 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909885 4959 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909889 4959 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909892 4959 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909895 4959 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909897 4959 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909902 4959 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:34:35.916122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909905 4959 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909907 4959 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909921 4959 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909924 4959 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909927 4959 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909931 4959 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909935 4959 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909938 4959 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909941 4959 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909944 4959 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909947 4959 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909950 4959 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909953 4959 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909956 4959 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909960 4959 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909962 4959 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909967 4959 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909970 4959 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909973 4959 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909975 4959 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909978 4959 flags.go:64] FLAG: --enable-server="true" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909981 4959 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909988 4959 flags.go:64] FLAG: --event-burst="100" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909991 4959 flags.go:64] FLAG: --event-qps="50" Oct 30 19:34:35.916685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909994 4959 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909996 4959 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.909999 4959 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910003 4959 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910007 4959 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910010 4959 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910012 4959 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910016 4959 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910019 4959 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910021 4959 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910024 4959 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910032 4959 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910035 4959 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910039 4959 flags.go:64] FLAG: --feature-gates="" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910043 4959 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910046 4959 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910049 4959 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910052 4959 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910054 4959 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910057 4959 flags.go:64] FLAG: --help="false" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910060 4959 flags.go:64] FLAG: --hostname-override="ip-10-0-132-115.ec2.internal" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910063 4959 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910066 4959 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:34:35.917302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910068 4959 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910072 4959 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910075 4959 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910077 4959 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910080 4959 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910083 4959 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910085 4959 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910088 4959 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910091 4959 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910094 4959 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910096 4959 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910099 4959 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910102 4959 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910105 4959 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910108 4959 flags.go:64] FLAG: --lock-file="" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910110 4959 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910114 4959 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910117 4959 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910122 4959 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910125 4959 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910128 4959 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910139 4959 flags.go:64] FLAG: --logging-format="text" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910143 4959 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910146 4959 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:34:35.917893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910149 4959 flags.go:64] FLAG: --manifest-url="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910152 4959 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910156 4959 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910159 4959 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910163 4959 flags.go:64] FLAG: --max-pods="110" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910166 4959 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910169 4959 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910172 4959 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910175 4959 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910177 4959 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910180 4959 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910183 4959 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910190 4959 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910193 4959 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910196 4959 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910199 4959 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910202 4959 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910204 4959 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910207 4959 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910210 4959 flags.go:64] FLAG: --port="10250" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910213 4959 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910216 4959 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0963631a1df966acd" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910219 4959 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910222 4959 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910225 4959 flags.go:64] FLAG: --register-node="true" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910228 4959 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:34:35.918474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910232 4959 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910234 4959 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910237 4959 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910240 4959 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910250 4959 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910253 4959 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910256 4959 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910259 4959 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910261 4959 flags.go:64] FLAG: --runonce="false" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910264 4959 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910267 4959 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910270 4959 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910273 4959 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910275 4959 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910278 4959 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910281 4959 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910284 4959 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910286 4959 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910289 4959 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910292 4959 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910295 4959 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910298 4959 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910300 4959 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910303 4959 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910308 4959 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:34:35.919206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910311 4959 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910314 4959 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910321 4959 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910324 4959 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910326 4959 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910329 4959 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910332 4959 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910335 4959 flags.go:64] FLAG: --v="2" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910339 4959 flags.go:64] FLAG: --version="false" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910343 4959 flags.go:64] FLAG: --vmodule="" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910346 4959 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.910351 4959 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910608 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910618 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910624 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910628 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910635 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910639 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910642 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910647 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910651 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910655 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.919881 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910659 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910663 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910668 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910672 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910679 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910685 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910692 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910696 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910700 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910704 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910708 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910712 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910716 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910720 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910724 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910728 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910732 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910736 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910740 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.920410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910744 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910748 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910757 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910761 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910772 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910776 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910781 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910784 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910790 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910795 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910800 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910804 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910809 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910813 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910817 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910821 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910826 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910830 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910833 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.921047 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910839 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910843 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910847 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910851 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910855 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910859 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910864 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910868 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910872 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910876 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910880 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910884 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910887 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910890 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910894 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910899 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910903 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910907 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910917 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910921 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.921681 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910925 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910929 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910933 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910936 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910940 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910944 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910949 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910953 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910957 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910961 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910965 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910969 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910974 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910978 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910982 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910986 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910990 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910994 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.910998 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.922337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911002 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911006 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911010 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911014 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911017 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911021 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911025 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911029 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911034 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911038 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911042 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911045 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911059 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911064 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911068 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911071 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911076 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911080 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.922834 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911083 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.911119 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911169 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911174 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911177 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911181 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911183 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911186 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911191 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911193 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911196 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911199 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911201 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911204 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911206 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.923279 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911214 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911217 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911220 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911222 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911225 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911227 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911229 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911233 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911235 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911238 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911240 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911243 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911250 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911253 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911256 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911258 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911261 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911263 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911266 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911268 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.923787 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911271 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911273 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911276 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911278 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911280 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911284 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911287 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911290 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911292 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911296 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911300 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911303 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911306 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911309 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911312 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911314 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911317 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911319 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911322 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.924402 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911325 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911327 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911331 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911333 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911336 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911338 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911346 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911349 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911351 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911354 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911356 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911359 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911361 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911365 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911369 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911371 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911374 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911376 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911380 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911383 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.924898 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911385 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911388 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911390 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911393 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911395 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911398 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911400 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911403 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911405 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911408 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911410 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911413 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911415 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911418 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911421 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911423 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911426 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911428 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911430 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.925368 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911438 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911441 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911443 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911446 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911448 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911450 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911453 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911455 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911458 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911460 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911462 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911468 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911470 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911473 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.911475 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.916647 4959 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.916663 4959 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.916680 4959 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.916685 4959 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916741 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.925843 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916747 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916751 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916754 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916757 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916759 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916762 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916765 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916767 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916772 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916776 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916780 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916783 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916786 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916788 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916791 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916794 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916796 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916799 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916802 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.926326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916804 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916807 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916810 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916813 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916817 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916820 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916823 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916826 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916829 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916831 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916833 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916836 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916838 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916841 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916843 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916846 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916848 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916851 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916853 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.926966 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916856 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916858 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916861 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916864 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916867 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916869 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916871 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916882 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916885 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916888 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916890 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916892 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916895 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916897 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916900 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916903 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916906 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916908 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916911 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916913 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.927410 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916916 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916919 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916921 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916924 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916926 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916929 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916931 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916934 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916936 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916939 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916941 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916943 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916946 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916948 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916951 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916953 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916956 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916959 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916961 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.927922 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916964 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916971 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916974 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916976 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916979 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916982 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916984 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916987 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916989 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916992 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916994 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916997 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.916999 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917002 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917004 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917007 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917009 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917012 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917014 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917017 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.928408 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917019 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917021 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917024 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917026 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917028 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917031 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917033 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917036 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.917063 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917201 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917206 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917209 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917211 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917214 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917218 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917220 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.929078 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917223 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917225 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917227 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917230 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917232 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917235 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917238 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917240 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917242 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917245 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917247 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917249 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917252 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917254 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917256 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917259 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917261 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917263 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917266 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.929712 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917268 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917270 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917273 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917275 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917278 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917280 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917283 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917285 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917288 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917290 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917293 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917295 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917298 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917301 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917303 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917306 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917308 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917311 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917313 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917316 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.930297 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917318 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917321 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917324 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917326 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917328 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917331 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917333 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917336 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917338 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917340 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917343 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917345 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917347 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917350 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917352 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917355 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917358 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917362 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917364 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.930809 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917367 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917369 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917372 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917374 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917377 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917379 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917382 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917384 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917389 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917393 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917397 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917399 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917402 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917405 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917408 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917411 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917413 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917416 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917418 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.931316 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917421 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917423 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917425 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917428 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917430 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917432 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917435 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917437 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917439 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917442 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917444 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917447 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917449 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917451 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917454 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917456 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917458 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917461 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917463 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917466 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.931836 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917469 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917472 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.917513 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917545 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917549 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917552 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917554 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917557 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917559 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917562 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917565 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917567 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917570 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917573 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917575 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917578 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:34:35.932326 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917580 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917582 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917585 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917587 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917590 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917592 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917594 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917597 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917600 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917602 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917605 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917607 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917610 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917612 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917614 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917617 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917619 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917624 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917626 4959 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:34:35.932810 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917629 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917631 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917634 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917636 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917638 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917641 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917643 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917646 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917648 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917651 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917653 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917655 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917658 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917660 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917663 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917665 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917668 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917670 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917672 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917675 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:34:35.933307 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917677 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917679 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917682 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917684 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917686 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917689 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917693 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917697 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917700 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917702 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917706 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917710 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917712 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917715 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917717 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917719 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917722 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917724 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:34:35.933812 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917728 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917731 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917734 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917736 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917739 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917741 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917743 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917746 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917748 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917751 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917753 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917755 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917758 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917760 4959 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917762 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917765 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917767 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917770 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917772 4959 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917774 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:34:35.934246 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917777 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917779 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917782 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917784 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917788 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917791 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917794 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917797 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917799 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917802 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917804 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917806 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917809 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917811 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917814 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:35.917816 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.917867 4959 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.918839 4959 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.918892 4959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:34:35.934780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.919069 4959 server.go:1014] "Starting client certificate rotation" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.919224 4959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.919276 4959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:28:55 +0000 UTC" deadline="2028-05-13 03:58:38.658139672 +0000 UTC" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.919310 4959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13448h24m2.73883138s" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.919512 4959 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.920606 4959 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.921727 4959 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.923344 4959 log.go:25] "Validated CRI v1 image API" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.924144 4959 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:34:35.935229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.925962 4959 fs.go:135] Filesystem UUIDs: map[701d8e5e-d4b4-4432-9c56-6384a7ad544a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7eda3421-4538-48a9-847b-4a0b59bdd0c0:/dev/nvme0n1p3] Oct 30 19:34:35.935476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.925980 4959 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/461942b33fa15a28c691eef8a5581c2881aa027ac0df8b987f3e6031407715f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/461942b33fa15a28c691eef8a5581c2881aa027ac0df8b987f3e6031407715f4/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d6ab31f712825ba8a00059f7273d53808e2eeebaa3bac54bd9f9dc186438e62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d6ab31f712825ba8a00059f7273d53808e2eeebaa3bac54bd9f9dc186438e62/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97e56f8114c184e3b1077fc75f28f433cdeaeb916fefe395e19864ce66611028/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97e56f8114c184e3b1077fc75f28f433cdeaeb916fefe395e19864ce66611028/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7232937f529ecdfecf34679747e36d3f76678ca6fe46f323caefb97aeb84048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7232937f529ecdfecf34679747e36d3f76678ca6fe46f323caefb97aeb84048/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5/volumes/kubernetes.io~projected/kube-api-access-8mtr9:{mountpoint:/var/lib/kubelet/pods/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5/volumes/kubernetes.io~projected/kube-api-access-8mtr9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16eeabd2-8f23-411a-b951-6207f95bdb43/volumes/kubernetes.io~projected/kube-api-access-cbxfk:{mountpoint:/var/lib/kubelet/pods/16eeabd2-8f23-411a-b951-6207f95bdb43/volumes/kubernetes.io~projected/kube-api-access-cbxfk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34d9f428-c41a-47ae-9ae5-9d93c76941ef/volumes/kubernetes.io~projected/kube-api-access-z6s4x:{mountpoint:/var/lib/kubelet/pods/34d9f428-c41a-47ae-9ae5-9d93c76941ef/volumes/kubernetes.io~projected/kube-api-access-z6s4x major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36088f70-cd5c-4911-ac09-cfb10529c20a/volumes/kubernetes.io~projected/kube-api-access-5v8dh:{mountpoint:/var/lib/kubelet/pods/36088f70-cd5c-4911-ac09-cfb10529c20a/volumes/kubernetes.io~projected/kube-api-access-5v8dh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d7a9e09-ccd3-4c01-92fd-41844e62b60f/volumes/kubernetes.io~projected/kube-api-access-wrhm7:{mountpoint:/var/lib/kubelet/pods/3d7a9e09-ccd3-4c01-92fd-41844e62b60f/volumes/kubernetes.io~projected/kube-api-access-wrhm7 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3da0ba8c-d364-4754-9069-47fe4a2017a6/volumes/kubernetes.io~projected/kube-api-access-pm7ps:{mountpoint:/var/lib/kubelet/pods/3da0ba8c-d364-4754-9069-47fe4a2017a6/volumes/kubernetes.io~projected/kube-api-access-pm7ps major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e7e7bcf-5f36-44d1-9232-167eebe939aa/volumes/kubernetes.io~projected/kube-api-access-gkx5r:{mountpoint:/var/lib/kubelet/pods/3e7e7bcf-5f36-44d1-9232-167eebe939aa/volumes/kubernetes.io~projected/kube-api-access-gkx5r major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc71932-8adb-4e3a-83e8-acfe87c1c896/volumes/kubernetes.io~projected/kube-api-access-xqwlv:{mountpoint:/var/lib/kubelet/pods/3fc71932-8adb-4e3a-83e8-acfe87c1c896/volumes/kubernetes.io~projected/kube-api-access-xqwlv major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~projected/kube-api-access-jr87n:{mountpoint:/var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~projected/kube-api-access-jr87n major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46fd767b-686b-42dd-acdd-5901294699ac/volumes/kubernetes.io~projected/kube-api-access-rk5m8:{mountpoint:/var/lib/kubelet/pods/46fd767b-686b-42dd-acdd-5901294699ac/volumes/kubernetes.io~projected/kube-api-access-rk5m8 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~projected/kube-api-access-qrvtz:{mountpoint:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~projected/kube-api-access-qrvtz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~projected/kube-api-access-zghxs:{mountpoint:/var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~projected/kube-api-access-zghxs major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~projected/kube-api-access-kjbvg:{mountpoint:/var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~projected/kube-api-access-kjbvg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c799558-73c8-45a2-83bf-dd99b24b8a36/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6c799558-73c8-45a2-83bf-dd99b24b8a36/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~projected/kube-api-access-llfbf:{mountpoint:/var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~projected/kube-api-access-llfbf major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cde5a04-1c75-44b3-8e49-bda52bb78768/volumes/kubernetes.io~projected/kube-api-access-zpv22:{mountpoint:/var/lib/kubelet/pods/7cde5a04-1c75-44b3-8e49-bda52bb78768/volumes/kubernetes.io~projected/kube-api-access-zpv22 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~projected/kube-api-access-hkknw:{mountpoint:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~projected/kube-api-access-hkknw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~projected/kube-api-access-llllb:{mountpoint:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~projected/kube-api-access-llllb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/kube-api-access-mp845:{mountpoint:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/kube-api-access-mp845 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~projected/kube-api-access-jf79p:{mountpoint:/var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~projected/kube-api-access-jf79p major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b001525d-fb68-4e3f-b967-f1a8d43f620b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b001525d-fb68-4e3f-b967-f1a8d43f620b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/kube-api-access-9mc5x:{mountpoint:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/kube-api-access-9mc5x major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e/volumes/kubernetes.io~projected/kube-api-access-8wsfb:{mountpoint:/var/lib/kubelet/pods/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e/volumes/kubernetes.io~projected/kube-api-access-8wsfb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdbd83b4-3edb-4b0b-aed1-27ca01f92641/volumes/kubernetes.io~projected/kube-api-access-85tsd:{mountpoint:/var/lib/kubelet/pods/cdbd83b4-3edb-4b0b-aed1-27ca01f92641/volumes/kubernetes.io~projected/kube-api-access-85tsd major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df601210-c5b1-4f3f-939b-bb8a11c3759d/volumes/kubernetes.io~projected/kube-api-access-nk4s5:{mountpoint:/var/lib/kubelet/pods/df601210-c5b1-4f3f-939b-bb8a11c3759d/volumes/kubernetes.io~projected/kube-api-access-nk4s5 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8699c88-8f03-4a7b-90f7-152a0118d517/volumes/kubernetes.io~projected/kube-api-access-bc7gn:{mountpoint:/var/lib/kubelet/pods/e8699c88-8f03-4a7b-90f7-152a0118d517/volumes/kubernetes.io~projected/kube-api-access-bc7gn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea3ab265-3e11-41d3-9192-e40c7588e1f5/volumes/kubernetes.io~projected/kube-api-access-g6t6g:{mountpoint:/var/lib/kubelet/pods/ea3ab265-3e11-41d3-9192-e40c7588e1f5/volumes/kubernetes.io~projected/kube-api-access-g6t6g major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20/volumes/kubernetes.io~projected/kube-api-access-xs2dh:{mountpoint:/var/lib/kubelet/pods/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20/volumes/kubernetes.io~projected/kube-api-access-xs2dh major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed54fa06-9be0-44dc-bf64-f48ea87b081f/volumes/kubernetes.io~projected/kube-api-access-wttzf:{mountpoint:/var/lib/kubelet/pods/ed54fa06-9be0-44dc-bf64-f48ea87b081f/volumes/kubernetes.io~projected/kube-api-access-wttzf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc0ef185-0820-4ae9-955b-0f35510667d9/volumes/kubernetes.io~projected/kube-api-access-7ql9v:{mountpoint:/var/lib/kubelet/pods/fc0ef185-0820-4ae9-955b-0f35510667d9/volumes/kubernetes.io~projected/kube-api-access-7ql9v major:0 minor:219 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/515c8e3af201e73a803639a7392cd7062d5cd59420a25f1d2425057ff280df49/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7a229df7ed5a40e47b6cd6e634ec963386cf9ad9ddd915da18ea6ad612d180be/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/001197cf4865dacb05cb8c5183d78a7c557b9ffd2dc9913c80b978539eb55f1b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ee912c9729ef0af8383ef2e44fe55bc3956368f5dfe8ca4e560b43a3c31e73a4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dd5d2168ed47dba37369415fcbc0c46a5277b8c0ea77fc422ded2735cf99d7e5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2753e7ece7aad994795041cd1f8f72cb9fbefed717363298d263588434665d26/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/aa5425c03707208e75f3999502617ba467647686aca3474aefb2bdd4db8cfad8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/73e13370c2d1f242cf3ef2f54badd8f542d1c5a62f6f0794de20e4dc26d7e02a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8c1bd331ed7e5d794c6b36c9e7847ac86a656c4a7f291366194e1fe3a64f02e7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ea4439178aa7ad1284399f4ee823e65b32714edb8ba9b5c298b14c77accddb60/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2bea387f959365eeb37078808dddb4c5276f99993ecb04002f326e38db1deaba/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/8d19c4134d331093880591fcce23713513950dd51cd2fd5c80ed34dc4cdfe8b6/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/49d3e4086e785fa1461df59e20a869c2a62c8bccb53a8464545990d9d3923b99/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0ef00b8ac3fc5b0ab96b871a8fe9490a762f61eb85074a5d29253b139e7fe063/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1451891a09c5c252896cb33fde41cde644aa02a15afe87e34961a0d31116f72a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/194198b566c942792eec04a21afa2924b30bdef3844058945c608cfbfefd56ab/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3ec67e018b3dfd6e9c9b70bb2d55f1b2f874bc75a0ab8ffadf76526ddbc420e9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6826bf3d5cf58194275d06c829da91dc9dd8ef4b08525326e455029ad65d2c8e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/125199a92e432bb3a0c952da5d470ef5a15a91faad8193251f10e06ef4e216b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/84e108de886015ab71c598d2f9f641d8ef886cdb835ef5f65ece15e1f80966b5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3715fe137efb5ee8bcf0a964354480e205c28adbc25301e1c01486450cbed8df/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/44880ef73546bb6bba8fb79297fa6e661e4fcf54cad8b3c495394defaafc703f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/1341bed6a1105f30e133422e9fe115884dd4dee6e2cf85b44a55a2fbeb6972a1/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/0c741fffda6b80b14d408b74cf9f8db92945cc86fe4df4b9e0eb3ef0756e050a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0aa9dfe05d1ea3e38d04666d01d37500c3fe57e0a5109fdfeb507526387ed962/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5244b7cf6ee99187389c3c5ce0672f1cffa99f51ec7c28ab2500a8cc955cd0fd/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1bca2356a7eb5e2d1e9e993c922e084497b64c5ed63eadd8ddbacd2252865b9d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1ecb2598f1b0eafc1953b20d79347f28eee801fd415eec04078d058a3fff028d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0b337099b1ac79b3083780726819396047307d01a2188f8937317fbaf37aebeb/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/348aa23ee23c6e1b44dbd273592dca853b2bcfb97d5a80269e700624ef6b28a7/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/ca87e14fe7d75a666749c678627f42f8a82852b820a739997f87f133da1b4005/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/280541396f01203dd27db857f4b3096a8e86fdad350d4a71a2c5aa17543218ca/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/5249fff3de9d69be36bd00d9cf943fcf1fa1add9558a98772bc02fc7f27e38d5/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/e2f434fee58f72074ff9a7c361aab7894f3b674eff9eb7997414db0642a33b0d/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/89ba409664559f7f8a08b1258ef7bf1a6e6e147896245fe4620ae330eed9b2f2/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/4d0d99a41d29015998d0d66457d3cdbfac5f6045d50079d7f9d6dcb4105aca99/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/9486721bf456e61c4df290a7da0c9d5ec9f9c74928658b7e2a949f5536370c79/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/8b4dda19da4c9f9e6aa23cc296e2fcd7528a21e0ed613e53ee10e07f8965d94e/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/c31010807f6b221942ad2c517e62992a262cc64a921cdc5febb1f39ff07cc592/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/2888cb67e75f8d2105b14a6e8cfbd8f5353aeabc8e4753811337031a1841df23/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/208471606c9469301aed1326ee0632028aeaef1756feb4e22189ea780b58c445/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/69fbcc7f209f45b6b68bf0a6bcb3efabe6f5ab080943516fc0b6fd5e97ac7594/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/39153cd68118cf313e376b530d87fcfe7cff153449a7e4ed7e0e63b99b95222d/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/b4d72f8fe2b01adc3394e7aecebecd876588c5134aa8b6b7e06bec5b1e806e59/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/a10ce93c5ced4c0c21d8c95d54c34406e3c2c297845c2137dcd529268bf59646/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/7013082fb9a16d5cab5687b3c98d6f82886559c0444a13a6e9787e819db32cae/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8be9b9354c5ccfdeeb9a362e657b341ea9637a777d3f75cea381655f446e0c7f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bce4c9585f03597080cd78200fe54b535e1d7db768cc2b535dcae058237018b3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c976d6a992f3e3d8b50aed695bdd949fb3231882669dc15812aaeaad79236521/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5dbaef3a16925be6544d55a4041a14731f4833300d8299785079f185ad6829b1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/064f864a9927952c9f01317c347e72437183cba668cc514038b0d4593ea81fc5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/95b8bb2597604bf0b06aaa4bd7747e2f9b7c4892f820e8fab823872123d37185/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c8e8934fd4d5be99742673109667d6f9aaeb0e6bab9aa4b850e4285ca85baefe/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/11a45f68889986832eb5e0e8bd942f0a0c3e68fff2692c4c5ec3d22d553769ed/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/48fb74d673561b02cddef5d1fe6142f71aaaf199b8c45ff2e5fcacf30ef98b86/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e1e7a757852dd95542f83fa3be071f87e1ae9d47162a0fd6a8feb21c4aa35e23/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:34:35.935977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.935612 4959 manager.go:217] Machine: {Timestamp:2026-10-30 19:34:35.935099297 +0000 UTC m=+0.064333209 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098721 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ad24fe900a4cc02effb48e64270d SystemUUID:ec23ad24-fe90-0a4c-c02e-ffb48e64270d BootID:b1a8c0b0-1a22-4cd4-8192-9fb03acad93d Filesystems:[{Device:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c74c97f92fc777cda5d6e7832702de9b04dafd13de07eaaf91b476033b85ae7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77274bb4581e8a1169344926c4082918154248ef853b80f506a4bbeff9291fb7/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~projected/kube-api-access-hkknw DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~projected/kube-api-access-qrvtz DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea3ab265-3e11-41d3-9192-e40c7588e1f5/volumes/kubernetes.io~projected/kube-api-access-g6t6g DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df601210-c5b1-4f3f-939b-bb8a11c3759d/volumes/kubernetes.io~projected/kube-api-access-nk4s5 DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~projected/kube-api-access-kjbvg DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cac704a6c47abe8c20ee56a15fcaf70cde9089601bc45d630e44d685d9c204d/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~projected/kube-api-access-zghxs DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16eeabd2-8f23-411a-b951-6207f95bdb43/volumes/kubernetes.io~projected/kube-api-access-cbxfk DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/587c1939e859fba3e0046b0767759b12634bc4aae80f4b11ab4a29fb8b72458e/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79105b2180e1fb525f517c133c9dc27ec223d811ce5750882136a8d0d3248b87/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d6ab31f712825ba8a00059f7273d53808e2eeebaa3bac54bd9f9dc186438e62/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/461942b33fa15a28c691eef8a5581c2881aa027ac0df8b987f3e6031407715f4/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/cdbd83b4-3edb-4b0b-aed1-27ca01f92641/volumes/kubernetes.io~projected/kube-api-access-85tsd DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~projected/kube-api-access-mp845 DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36088f70-cd5c-4911-ac09-cfb10529c20a/volumes/kubernetes.io~projected/kube-api-access-5v8dh DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e/volumes/kubernetes.io~projected/kube-api-access-8wsfb DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ef91e24f1131ddeaa82d09cc4b863e30822e8be15215f8fcc3c820b254c5bbd/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5425e2a15b7b32187ecf6e1c974853e1bef4dd238e27f05a3d485a1004c61cf7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7232937f529ecdfecf34679747e36d3f76678ca6fe46f323caefb97aeb84048/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30ac5ae9f2c36ff917ee747adef7187d08fd92eb26f96bafcda0e65dd149a6d8/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3da0ba8c-d364-4754-9069-47fe4a2017a6/volumes/kubernetes.io~projected/kube-api-access-pm7ps DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc71932-8adb-4e3a-83e8-acfe87c1c896/volumes/kubernetes.io~projected/kube-api-access-xqwlv DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34d9f428-c41a-47ae-9ae5-9d93c76941ef/volumes/kubernetes.io~projected/kube-api-access-z6s4x DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e8699c88-8f03-4a7b-90f7-152a0118d517/volumes/kubernetes.io~projected/kube-api-access-bc7gn DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54bc4737fd19813ffad44b6e9e7d16f2b208f20d5b164810700face406fb0d2d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97e56f8114c184e3b1077fc75f28f433cdeaeb916fefe395e19864ce66611028/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dcadcb7-1c64-4956-bf5c-489946be6487/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6c85830-f826-4df6-a48f-372c870e653a/volumes/kubernetes.io~projected/kube-api-access-llllb DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc0ef185-0820-4ae9-955b-0f35510667d9/volumes/kubernetes.io~projected/kube-api-access-7ql9v DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~projected/kube-api-access-jr87n DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6603795ae613944c600c784fb346e68ccc470bcf68a3fd17727a76566eadfbf/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c54f44a6be18c5c6458e9adb9d569f7f4043da6dea8b56e616ce7237dc4daf79/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~projected/kube-api-access-jf79p DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/46fd767b-686b-42dd-acdd-5901294699ac/volumes/kubernetes.io~projected/kube-api-access-rk5m8 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d21768a46212f7f2d9a8dbe0e191bbcfeed937534e47f4ed25b5d87b701c453b/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98b9bf6ad247d0bf96ee71b127115acb6870dc58246308cd39fb5d853f1ca8d/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed54fa06-9be0-44dc-bf64-f48ea87b081f/volumes/kubernetes.io~projected/kube-api-access-wttzf DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3bc7ac281f4ecdcc82abd3f3429178724f7d55a2977aa4a00ac548656668a51/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20/volumes/kubernetes.io~projected/kube-api-access-xs2dh DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/volumes/kubernetes.io~projected/kube-api-access-llfbf DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d7a9e09-ccd3-4c01-92fd-41844e62b60f/volumes/kubernetes.io~projected/kube-api-access-wrhm7 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/410e9153-45cc-485e-ba92-efa60cca301e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/498b6873-9b05-4819-8d44-c6bbab466828/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b001525d-fb68-4e3f-b967-f1a8d43f620b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5/volumes/kubernetes.io~projected/kube-api-access-8mtr9 DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/869f969b38eaa6302e6c477efdfa42f2435a7737098459082cbf148736527e99/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79a20541cb5a9bed2d97bf8df47d5f86f02692463d545d9279ea47d525c1bd8b/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c799558-73c8-45a2-83bf-dd99b24b8a36/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes/kubernetes.io~projected/kube-api-access-9mc5x DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef89f3c592ef20d916344a90e437984542a518cae0fe21245ca2d23f11713b85/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7cde5a04-1c75-44b3-8e49-bda52bb78768/volumes/kubernetes.io~projected/kube-api-access-zpv22 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae1ef3c9-11c5-48c2-b1db-166270b55b09/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3e7e7bcf-5f36-44d1-9232-167eebe939aa/volumes/kubernetes.io~projected/kube-api-access-gkx5r DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0cac704a6c47abe MacAddress:76:66:1f:71:df:07 Speed:10000 Mtu:8901} {Name:30ac5ae9f2c36ff MacAddress:ee:83:77:56:3d:db Speed:10000 Mtu:8901} {Name:461942b33fa15a2 MacAddress:36:c1:c2:1a:40:e7 Speed:10000 Mtu:8901} {Name:5425e2a15b7b321 MacAddress:16:be:96:72:50:ac Speed:10000 Mtu:8901} {Name:587c1939e859fba MacAddress:56:ec:c7:15:38:b8 Speed:10000 Mtu:8901} {Name:79a20541cb5a9be MacAddress:6e:2d:c9:28:5f:22 Speed:10000 Mtu:8901} {Name:7d6ab31f712825b MacAddress:0e:7e:7d:95:54:2d Speed:10000 Mtu:8901} {Name:97e56f8114c184e MacAddress:ee:08:99:99:cb:ca Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cd:d8:de:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:da:1e:33:bb:2c:27 Speed:0 Mtu:8901} {Name:c7232937f529ecd MacAddress:f6:23:fe:f9:71:ad Speed:10000 Mtu:8901} {Name:d21768a46212f7f MacAddress:ca:70:6d:2f:ee:5c Speed:10000 Mtu:8901} {Name:e98b9bf6ad247d0 MacAddress:46:10:a4:88:45:64 Speed:10000 Mtu:8901} {Name:ef89f3c592ef20d MacAddress:1a:84:29:c5:09:6e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cd:d8:de:bf Speed:0 Mtu:9001} {Name:f3bc7ac281f4ecd MacAddress:2a:2d:57:a5:35:66 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:52:f0:fd:9b:89:39 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:ae:48:c0:31:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:34:35.936152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.935981 4959 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:34:35.936152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936023 4959 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:34:35.936318 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936292 4959 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:34:35.936439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936319 4959 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:34:35.936480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936442 4959 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:34:35.936480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936449 4959 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:34:35.936480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936459 4959 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:34:35.936480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936476 4959 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:34:35.936605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936511 4959 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:34:35.936665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936655 4959 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:34:35.936782 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936774 4959 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:34:35.936811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936807 4959 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:34:35.936838 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936816 4959 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:34:35.936838 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936830 4959 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:34:35.936890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936840 4959 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:34:35.936890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.936850 4959 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:34:35.937879 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.937864 4959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:34:35.937927 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.937882 4959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:34:35.938386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.938371 4959 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:34:35.940958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.940941 4959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:34:35.940958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.940955 4959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:34:35.941110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941096 4959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:34:35.941332 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941316 4959 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:34:35.941404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941357 4959 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:34:35.941511 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941482 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:34:35.941568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941518 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:34:35.941568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941529 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:34:35.941568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941539 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:34:35.941568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941548 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:34:35.941568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941567 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941576 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941586 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941596 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941605 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941624 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941639 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941690 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:34:35.941803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.941701 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:34:35.945117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.945101 4959 server.go:1263] "Started kubelet" Oct 30 19:34:35.945202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.945154 4959 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:34:35.945249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.945196 4959 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:34:35.945288 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.945270 4959 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:34:35.945725 ip-10-0-132-115 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:34:35.949017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.947311 4959 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:34:35.961579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.961558 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:35.964510 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964478 4959 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:34:35.964601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964563 4959 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:34:35.964601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964582 4959 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:34:35.964700 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:35.964610 4959 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Oct 30 19:34:35.964781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964569 4959 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:34:35.964844 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964834 4959 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:34:35.966260 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966241 4959 factory.go:55] Registering systemd factory Oct 30 19:34:35.966260 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966262 4959 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:34:35.966925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966837 4959 factory.go:153] Registering CRI-O factory Oct 30 19:34:35.966925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966856 4959 factory.go:223] Registration of the crio container factory successfully Oct 30 19:34:35.966925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.964549 4959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:34:35.967373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966971 4959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:28:57 +0000 UTC" deadline="2028-07-28 20:06:51.315495328 +0000 UTC" Oct 30 19:34:35.967373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.966990 4959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15288h32m15.348508423s" Oct 30 19:34:35.967373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.967213 4959 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:34:35.967373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.967234 4959 factory.go:103] Registering Raw factory Oct 30 19:34:35.967373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.967253 4959 manager.go:1199] Started watching for new ooms in manager Oct 30 19:34:35.967654 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.967639 4959 manager.go:319] Starting recovery of all containers Oct 30 19:34:35.968110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968056 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7a9e09-ccd3-4c01-92fd-41844e62b60f" volumeName="kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968115 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46fd767b-686b-42dd-acdd-5901294699ac" volumeName="kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968131 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c799558-73c8-45a2-83bf-dd99b24b8a36" volumeName="kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968144 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" volumeName="kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968156 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7bc2f1-d85e-4656-b3a2-ca8c46710e20" volumeName="kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968168 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c85830-f826-4df6-a48f-372c870e653a" volumeName="kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968181 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374" volumeName="kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968192 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcadcb7-1c64-4956-bf5c-489946be6487" volumeName="kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968205 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates" seLinuxMountContext="" Oct 30 19:34:35.968213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968217 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df601210-c5b1-4f3f-939b-bb8a11c3759d" volumeName="kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968228 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8699c88-8f03-4a7b-90f7-152a0118d517" volumeName="kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968239 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" volumeName="kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968253 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c85830-f826-4df6-a48f-372c870e653a" volumeName="kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968264 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968276 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8699c88-8f03-4a7b-90f7-152a0118d517" volumeName="kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968290 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea3ab265-3e11-41d3-9192-e40c7588e1f5" volumeName="kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968301 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04a17b98-bb85-4bde-b1b4-29d1b39ad1c5" volumeName="kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968312 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc71932-8adb-4e3a-83e8-acfe87c1c896" volumeName="kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968323 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="498b6873-9b05-4819-8d44-c6bbab466828" volumeName="kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968334 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27c42b-524e-4f97-b89b-87047a1d32f7" volumeName="kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968345 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968356 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34d9f428-c41a-47ae-9ae5-9d93c76941ef" volumeName="kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968367 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="410e9153-45cc-485e-ba92-efa60cca301e" volumeName="kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968377 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968389 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c85830-f826-4df6-a48f-372c870e653a" volumeName="kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968400 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1ef3c9-11c5-48c2-b1db-166270b55b09" volumeName="kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968410 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e" volumeName="kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968422 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54fa06-9be0-44dc-bf64-f48ea87b081f" volumeName="kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968438 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54fa06-9be0-44dc-bf64-f48ea87b081f" volumeName="kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968449 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7a9e09-ccd3-4c01-92fd-41844e62b60f" volumeName="kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968461 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="498b6873-9b05-4819-8d44-c6bbab466828" volumeName="kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968487 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcadcb7-1c64-4956-bf5c-489946be6487" volumeName="kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968517 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968537 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54fa06-9be0-44dc-bf64-f48ea87b081f" volumeName="kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968548 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b001525d-fb68-4e3f-b967-f1a8d43f620b" volumeName="kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968560 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04a17b98-bb85-4bde-b1b4-29d1b39ad1c5" volumeName="kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968572 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3da0ba8c-d364-4754-9069-47fe4a2017a6" volumeName="kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968583 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7e7bcf-5f36-44d1-9232-167eebe939aa" volumeName="kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968594 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7e7bcf-5f36-44d1-9232-167eebe939aa" volumeName="kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968605 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968615 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" volumeName="kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968627 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968640 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="410e9153-45cc-485e-ba92-efa60cca301e" volumeName="kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968651 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c799558-73c8-45a2-83bf-dd99b24b8a36" volumeName="kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968664 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cde5a04-1c75-44b3-8e49-bda52bb78768" volumeName="kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968675 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c85830-f826-4df6-a48f-372c870e653a" volumeName="kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968686 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968699 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968712 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968724 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968735 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e" volumeName="kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968746 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7bc2f1-d85e-4656-b3a2-ca8c46710e20" volumeName="kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968758 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc0ef185-0820-4ae9-955b-0f35510667d9" volumeName="kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968769 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dcadcb7-1c64-4956-bf5c-489946be6487" volumeName="kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968780 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968792 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab02b8d2-0815-412a-a732-ef750ca1130b" volumeName="kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968812 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdbd83b4-3edb-4b0b-aed1-27ca01f92641" volumeName="kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968823 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16eeabd2-8f23-411a-b951-6207f95bdb43" volumeName="kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk" seLinuxMountContext="" Oct 30 19:34:35.968805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968835 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="498b6873-9b05-4819-8d44-c6bbab466828" volumeName="kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968847 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968865 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdbd83b4-3edb-4b0b-aed1-27ca01f92641" volumeName="kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968877 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374" volumeName="kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968888 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968899 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d7a9e09-ccd3-4c01-92fd-41844e62b60f" volumeName="kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968914 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="410e9153-45cc-485e-ba92-efa60cca301e" volumeName="kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968926 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" volumeName="kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968940 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968950 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968961 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54fa06-9be0-44dc-bf64-f48ea87b081f" volumeName="kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968974 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1ef3c9-11c5-48c2-b1db-166270b55b09" volumeName="kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968985 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b936af65-9699-44af-bbec-bd298c20d1fc" volumeName="kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.968996 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36088f70-cd5c-4911-ac09-cfb10529c20a" volumeName="kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969008 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374" volumeName="kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969020 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969031 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8699c88-8f03-4a7b-90f7-152a0118d517" volumeName="kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969043 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cde5a04-1c75-44b3-8e49-bda52bb78768" volumeName="kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969055 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c85830-f826-4df6-a48f-372c870e653a" volumeName="kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969066 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" volumeName="kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots" seLinuxMountContext="" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969078 4959 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:34:35.970800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.969085 4959 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:34:35.980273 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.980255 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:35.987254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.986549 4959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:34:35.988561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.988166 4959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:34:35.988561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.988188 4959 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:34:35.988561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.988208 4959 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:34:35.988561 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:35.988246 4959 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:34:35.992669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.992645 4959 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:35.992903 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.992863 4959 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:35.996962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.996938 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:35.997068 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:35.996973 4959 generic.go:356] "Generic (PLEG): container finished" podID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" exitCode=255 Oct 30 19:34:36.004255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.004232 4959 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:36.004355 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.004295 4959 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:34:36.004797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.004763 4959 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:34:36.005045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.005024 4959 generic.go:356] "Generic (PLEG): container finished" podID="47397ab791596d97226855f409f94517" containerID="4fb178cb2f35da6ca3652d465c0ed33cad4adc85cc4be13d74a644de0656b989" exitCode=0 Oct 30 19:34:36.013145 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013118 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="3d9c6dd57e2efbc01de05b0a6662772e5fee7414ce76ce4ee056479de7f62a94" exitCode=0 Oct 30 19:34:36.013145 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013143 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="f6689b11f01b57805e62ab974f892e223a96ab993c4c5da6a5d622303757b4cc" exitCode=0 Oct 30 19:34:36.013276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013153 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="b7f2cae624f18dec22e835538d8aebcb964833d55bf7d11e17809979f9b4687c" exitCode=0 Oct 30 19:34:36.013276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013161 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="1ca0891ba6e95bcd534028e75d7036edddabeda66d113b9956da45b828fa1048" exitCode=0 Oct 30 19:34:36.013276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013169 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="7be4539c7b327565785520a4c98cf5275726db0fff93d09ec4223d8c92fcdca3" exitCode=0 Oct 30 19:34:36.013276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.013177 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed54fa06-9be0-44dc-bf64-f48ea87b081f" containerID="41dcdd09319759ad0198ae4cf8863295945614d36a7e4ad739877fa3e86c25dd" exitCode=0 Oct 30 19:34:36.056753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.056735 4959 manager.go:324] Recovery completed Oct 30 19:34:36.076396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076336 4959 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:34:36.076396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076361 4959 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:34:36.076396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076382 4959 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:34:36.076603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076576 4959 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:34:36.076648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076592 4959 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:34:36.076648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076623 4959 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:34:36.076648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076631 4959 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:34:36.076648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076639 4959 policy_none.go:50] "Start" Oct 30 19:34:36.076826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076658 4959 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:34:36.076826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076671 4959 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:34:36.076915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076901 4959 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:34:36.076915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076912 4959 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:34:36.076979 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.076919 4959 policy_none.go:44] "Start" Oct 30 19:34:36.081374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.081349 4959 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:34:36.084938 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.084923 4959 manager.go:341] "Starting Device Plugin manager" Oct 30 19:34:36.085641 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:36.084956 4959 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:34:36.085695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.085647 4959 server.go:87] "Starting device plugin registration server" Oct 30 19:34:36.085970 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.085958 4959 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:34:36.086003 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.085975 4959 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:34:36.086090 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.086074 4959 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:34:36.086187 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.086156 4959 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:34:36.086263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.086197 4959 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:34:36.086263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.086203 4959 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:34:36.086335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.086256 4959 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:34:36.086185298Z","UUID":"6c8d3633-6f63-4cc8-94f1-858bd3520709","Handler":null,"Name":"","Endpoint":""} Oct 30 19:34:36.087630 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.087605 4959 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:34:36.087630 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.087636 4959 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:34:36.087797 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:36.087779 4959 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:34:36.088476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.088435 4959 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal"] Oct 30 19:34:36.089224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089185 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerStarted","Data":"3cf8e869ec2131b7ff082c9af5a124b213e7953c40fb35c2a02e4559520bb718"} Oct 30 19:34:36.089309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089236 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerDied","Data":"4fb178cb2f35da6ca3652d465c0ed33cad4adc85cc4be13d74a644de0656b989"} Oct 30 19:34:36.089309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089259 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"47397ab791596d97226855f409f94517","Type":"ContainerStarted","Data":"86811f72a555b415b7b8abf33dc4fd1a441de47e8d7570da9fcc3777965e9543"} Oct 30 19:34:36.089414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089311 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"4bb73836ec8ec4c4107421909743aa5c","Type":"ContainerStarted","Data":"6aefa0d98003385c45ed51eb15da5589dafa8e2fdc5c005ff6fbcb760b44229d"} Oct 30 19:34:36.089414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089336 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"4bb73836ec8ec4c4107421909743aa5c","Type":"ContainerStarted","Data":"913c7d64fce0bcbf8dcf05f6e1d5bca53a845164cc71912bc10f2a329408b320"} Oct 30 19:34:36.089414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089350 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.089414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.089401 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.110649 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.110626 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:36.110740 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:36.110665 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.111186 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.111170 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:36.111258 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:36.111204 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.186964 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.186928 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:34:36.188197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.188181 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Oct 30 19:34:36.188298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.188209 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 19:34:36.188298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.188219 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Oct 30 19:34:36.188298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.188255 4959 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.204849 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.204824 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.204933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.204866 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.204933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.204912 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.219417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.219402 4959 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.219462 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.219449 4959 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305047 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305085 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305101 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305147 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4bb73836ec8ec4c4107421909743aa5c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"4bb73836ec8ec4c4107421909743aa5c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305153 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.305240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.305188 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47397ab791596d97226855f409f94517-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"47397ab791596d97226855f409f94517\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:36.937443 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:36.937414 4959 apiserver.go:52] "Watching apiserver" Oct 30 19:34:37.008448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.008408 4959 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:34:37.009113 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.009089 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk","openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv","openshift-ingress-canary/ingress-canary-r65qm","openshift-image-registry/node-ca-fr4xc","openshift-ingress/router-default-fd9894b6f-skqbz","openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl","openshift-network-console/networking-console-plugin-86c478bb88-mb2fw","kube-system/global-pull-secret-syncer-shwzj","kube-system/kas-connection-checker-685895c7c9-7t7rs","kube-system/kas-connection-checker-685895c7c9-kv78f","openshift-insights/insights-operator-7c4986d449-frzlq","openshift-insights/insights-runtime-extractor-q8jlb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-k7rbb","openshift-dns/node-resolver-jw9s8","openshift-image-registry/image-registry-6775f6f89c-7npjm","openshift-network-operator/iptables-alerter-przhx","kube-system/konnectivity-agent-6bzkf","kube-system/kas-connection-checker-685895c7c9-h4vsd","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal","openshift-network-diagnostics/network-check-target-8f2fj","openshift-cluster-node-tuning-operator/tuned-qzmns","openshift-monitoring/cluster-monitoring-operator-884576c57-5484c","openshift-multus/network-metrics-daemon-57wdt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5","openshift-multus/multus-l5hwp","openshift-console-operator/console-operator-7cd56b8dbc-zl8sn","openshift-image-registry/image-registry-755b5c4488-4xx52","openshift-ovn-kubernetes/ovnkube-node-pfvgl","openshift-dns/dns-default-sg9nh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz"] Oct 30 19:34:37.010351 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.010331 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.011396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.011378 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.011753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.011728 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.013700 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.013679 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.014048 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.014031 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.023019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.023002 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.023478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.023464 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.023574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.023521 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.023633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.023614 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.024126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.024113 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:34:37.024655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.024636 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.024742 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.024643 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:34:37.024796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.024781 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.025907 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.025890 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.027358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.027343 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:34:37.045805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.045785 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:37.045957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.045943 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:37.066945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.066923 4959 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:34:37.067080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.067067 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:37.067131 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.067108 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Oct 30 19:34:37.067576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.067563 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:34:37.067638 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.067608 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Oct 30 19:34:37.108035 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108011 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.108184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108040 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.108184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108067 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsfb\" (UniqueName: \"kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.108184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108094 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.108184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108120 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.108345 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108205 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.108345 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108229 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.108345 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108276 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.108345 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108306 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108354 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108389 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf79p\" (UniqueName: \"kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108411 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108427 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108449 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.108537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108486 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108556 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108587 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cde5a04-1c75-44b3-8e49-bda52bb78768-tmp-dir\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108589 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-tuned\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108661 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108739 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tsd\" (UniqueName: \"kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.108796 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108770 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.108978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108922 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.108978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108942 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.108978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.108960 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.109079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109024 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7gn\" (UniqueName: \"kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.109079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-tmp\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.109079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.109079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109066 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109100 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109140 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109157 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109172 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109197 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr87n\" (UniqueName: \"kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.109257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109232 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zghxs\" (UniqueName: \"kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109265 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109299 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109362 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109390 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttzf\" (UniqueName: \"kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109415 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109447 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109489 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109551 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.109621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109577 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109605 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109657 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109685 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109712 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk5m8\" (UniqueName: \"kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8\") pod \"network-check-source-6f78d59f99-l2ccl\" (UID: \"46fd767b-686b-42dd-acdd-5901294699ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109738 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109772 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109802 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109829 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109847 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109889 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109927 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109956 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llllb\" (UniqueName: \"kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.109982 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110010 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110083 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110107 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110157 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110197 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110214 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110233 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2dh\" (UniqueName: \"kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110248 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110264 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ql9v\" (UniqueName: \"kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v\") pod \"kas-connection-checker-685895c7c9-h4vsd\" (UID: \"fc0ef185-0820-4ae9-955b-0f35510667d9\") " pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110364 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110391 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7ps\" (UniqueName: \"kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps\") pod \"volume-data-source-validator-58c9f5f46d-ppsxz\" (UID: \"3da0ba8c-d364-4754-9069-47fe4a2017a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110408 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6t6g\" (UniqueName: \"kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110424 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110439 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.110478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110460 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110511 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110559 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110587 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110561 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110615 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110641 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110668 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110680 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-default-certificate\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110689 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dcadcb7-1c64-4956-bf5c-489946be6487-tmp\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110725 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110751 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s4x\" (UniqueName: \"kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x\") pod \"migrator-66df5d496b-sddsk\" (UID: \"34d9f428-c41a-47ae-9ae5-9d93c76941ef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110766 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110782 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110890 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110909 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-data-volume\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.110946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.110924 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111023 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111053 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111075 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111109 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpv22\" (UniqueName: \"kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111143 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhm7\" (UniqueName: \"kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111178 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111207 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111227 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111313 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111346 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111371 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111442 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111460 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.111518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111477 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkknw\" (UniqueName: \"kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111546 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111569 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxfk\" (UniqueName: \"kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk\") pod \"kas-connection-checker-685895c7c9-7t7rs\" (UID: \"16eeabd2-8f23-411a-b951-6207f95bdb43\") " pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111682 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfbf\" (UniqueName: \"kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111703 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx5r\" (UniqueName: \"kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111736 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111762 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111803 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111830 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111862 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111915 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.112020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.111948 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112044 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112084 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112117 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112141 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112146 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-tmp-dir\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112167 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112234 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112266 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112345 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112362 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.112439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112434 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvtz\" (UniqueName: \"kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112534 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112564 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112591 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8dh\" (UniqueName: \"kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112618 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112670 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-snapshots\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112718 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112738 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112765 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112784 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk4s5\" (UniqueName: \"kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5\") pod \"kas-connection-checker-685895c7c9-kv78f\" (UID: \"df601210-c5b1-4f3f-939b-bb8a11c3759d\") " pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112800 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112826 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtr9\" (UniqueName: \"kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.112866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112844 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112883 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112912 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.112946 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113014 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113056 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113168 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbvg\" (UniqueName: \"kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113193 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113217 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113249 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113385 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.113470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113440 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.113826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113475 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.113826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113528 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.113826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113553 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.113826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.113573 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.124998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.124979 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.125147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125130 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:34:37.125342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125326 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.125402 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125368 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:34:37.125402 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125373 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.125402 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125388 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:34:37.125539 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125403 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.125539 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125427 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.125712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125698 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.125849 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125837 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:34:37.125881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125852 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.125920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125894 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.125986 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.125973 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126084 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126071 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126084 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126078 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.126198 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126103 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.126198 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126143 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.126258 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126234 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:34:37.126327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126312 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:34:37.126386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126325 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:34:37.126386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126319 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126337 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.126386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126377 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:34:37.126386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126382 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126461 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.126568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126478 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126578 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:34:37.126759 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126747 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:34:37.126930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126916 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.126930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.126923 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.127181 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127166 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:34:37.127239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127187 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:34:37.127239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127194 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.127239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127191 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.127239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127199 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:34:37.127369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127240 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.127369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127251 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.127369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127281 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.127369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127312 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.127957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127936 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:34:37.128041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.127996 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.128041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.128010 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.128708 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.128690 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:34:37.129011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.128984 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.129077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129011 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.129077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129055 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.129077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129072 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.129217 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129190 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsfb\" (UniqueName: \"kubernetes.io/projected/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-kube-api-access-8wsfb\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.129336 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129321 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.129384 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129373 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:34:37.129477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129458 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-env-overrides\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.129625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129458 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e27c42b-524e-4f97-b89b-87047a1d32f7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.129683 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129656 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.129755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129738 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:34:37.129820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129802 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:34:37.129994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.129980 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:34:37.130077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130060 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:34:37.130209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130196 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:34:37.130257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130239 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:34:37.130401 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130384 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.130475 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130462 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-serviceca\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.130598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130481 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-config\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.130665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130649 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/410e9153-45cc-485e-ba92-efa60cca301e-config\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.130713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130667 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-config\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.130954 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.130937 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-daemon-config\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.131172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.131155 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c799558-73c8-45a2-83bf-dd99b24b8a36-konnectivity-ca\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:37.132085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132063 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-serving-cert\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.132173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-config-volume\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.132221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132179 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.132221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132182 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.132221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132151 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-service-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.132713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-serving-cert\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.132751 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.132713 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:34:37.133411 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.133397 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8699c88-8f03-4a7b-90f7-152a0118d517-cni-binary-copy\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.133451 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.133410 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.133546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.133532 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.134727 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.134703 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:34:37.134727 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.134718 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.136118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.136100 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:34:37.136657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.136642 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:34:37.136720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.136660 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:34:37.137387 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137373 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:34:37.137622 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137609 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:34:37.137702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137610 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:34:37.137702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137615 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:34:37.137813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137702 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:34:37.137813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.137702 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:34:37.138210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138190 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf79p\" (UniqueName: \"kubernetes.io/projected/ae1ef3c9-11c5-48c2-b1db-166270b55b09-kube-api-access-jf79p\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:37.138325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138311 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9pzwv\"" Oct 30 19:34:37.138419 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138408 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:34:37.138481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138428 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l9ldq\"" Oct 30 19:34:37.138722 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138709 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:34:37.138788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138746 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fkztb\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138886 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9nnsl\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138894 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9cqt\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138886 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vj26l\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138916 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138926 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4mr2r\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138929 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zv66j\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138953 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4c78\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138958 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqfcc\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138956 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xhvhw\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138963 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lhsbg\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.138972 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-qv89v\"" Oct 30 19:34:37.139001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139003 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2hnnk\"" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139073 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tsd\" (UniqueName: \"kubernetes.io/projected/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-kube-api-access-85tsd\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139163 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-52dmm\"" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139198 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxzn8\"" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139204 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6fz68\"" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139289 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mtjn\"" Oct 30 19:34:37.139390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139387 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk4qf\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139449 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-khwrx\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139471 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139522 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hgf7z\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139531 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p87n7\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139539 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lkchj\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139539 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sqtpr\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139542 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-m8v7z\"" Oct 30 19:34:37.139691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.139651 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-iptables-alerter-script\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.140823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.140805 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.140860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.140827 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1ef3c9-11c5-48c2-b1db-166270b55b09-cert\") pod \"ingress-canary-r65qm\" (UID: \"ae1ef3c9-11c5-48c2-b1db-166270b55b09\") " pod="openshift-ingress-canary/ingress-canary-r65qm" Oct 30 19:34:37.140860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.140808 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.141931 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.141916 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/410e9153-45cc-485e-ba92-efa60cca301e-serving-cert\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.143119 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.143100 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.143152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.143124 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c799558-73c8-45a2-83bf-dd99b24b8a36-agent-certs\") pod \"konnectivity-agent-6bzkf\" (UID: \"6c799558-73c8-45a2-83bf-dd99b24b8a36\") " pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:37.143152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.143133 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-stats-auth\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.143206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.143137 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c85830-f826-4df6-a48f-372c870e653a-ovn-node-metrics-cert\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.144101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144086 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.144159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144149 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:34:37.144196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144179 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:34:37.144738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144712 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ql9v\" (UniqueName: \"kubernetes.io/projected/fc0ef185-0820-4ae9-955b-0f35510667d9-kube-api-access-7ql9v\") pod \"kas-connection-checker-685895c7c9-h4vsd\" (UID: \"fc0ef185-0820-4ae9-955b-0f35510667d9\") " pod="kube-system/kas-connection-checker-685895c7c9-h4vsd" Oct 30 19:34:37.144822 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144761 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7gn\" (UniqueName: \"kubernetes.io/projected/e8699c88-8f03-4a7b-90f7-152a0118d517-kube-api-access-bc7gn\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.144992 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.144973 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkknw\" (UniqueName: \"kubernetes.io/projected/9dcadcb7-1c64-4956-bf5c-489946be6487-kube-api-access-hkknw\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.147897 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2dh\" (UniqueName: \"kubernetes.io/projected/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-api-access-xs2dh\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.147907 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llllb\" (UniqueName: \"kubernetes.io/projected/a6c85830-f826-4df6-a48f-372c870e653a-kube-api-access-llllb\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148102 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6t6g\" (UniqueName: \"kubernetes.io/projected/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kube-api-access-g6t6g\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148109 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk5m8\" (UniqueName: \"kubernetes.io/projected/46fd767b-686b-42dd-acdd-5901294699ac-kube-api-access-rk5m8\") pod \"network-check-source-6f78d59f99-l2ccl\" (UID: \"46fd767b-686b-42dd-acdd-5901294699ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l2ccl" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148118 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148117 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.148177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148155 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpv22\" (UniqueName: \"kubernetes.io/projected/7cde5a04-1c75-44b3-8e49-bda52bb78768-kube-api-access-zpv22\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.148598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148239 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttzf\" (UniqueName: \"kubernetes.io/projected/ed54fa06-9be0-44dc-bf64-f48ea87b081f-kube-api-access-wttzf\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.148598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148396 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr87n\" (UniqueName: \"kubernetes.io/projected/410e9153-45cc-485e-ba92-efa60cca301e-kube-api-access-jr87n\") pod \"service-ca-operator-544d749c8c-65x2c\" (UID: \"410e9153-45cc-485e-ba92-efa60cca301e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" Oct 30 19:34:37.148704 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148624 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zghxs\" (UniqueName: \"kubernetes.io/projected/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-kube-api-access-zghxs\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.148704 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148660 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5\" (UID: \"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" Oct 30 19:34:37.148820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148729 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:34:37.149017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.148948 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx5r\" (UniqueName: \"kubernetes.io/projected/3e7e7bcf-5f36-44d1-9232-167eebe939aa-kube-api-access-gkx5r\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.149103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149087 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhm7\" (UniqueName: \"kubernetes.io/projected/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-kube-api-access-wrhm7\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.149169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149125 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3e7e7bcf-5f36-44d1-9232-167eebe939aa-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.149169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149148 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxfk\" (UniqueName: \"kubernetes.io/projected/16eeabd2-8f23-411a-b951-6207f95bdb43-kube-api-access-cbxfk\") pod \"kas-connection-checker-685895c7c9-7t7rs\" (UID: \"16eeabd2-8f23-411a-b951-6207f95bdb43\") " pod="kube-system/kas-connection-checker-685895c7c9-7t7rs" Oct 30 19:34:37.149276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149171 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b001525d-fb68-4e3f-b967-f1a8d43f620b-original-pull-secret\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.149363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149343 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfbf\" (UniqueName: \"kubernetes.io/projected/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-kube-api-access-llfbf\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.149608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149579 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8dh\" (UniqueName: \"kubernetes.io/projected/36088f70-cd5c-4911-ac09-cfb10529c20a-kube-api-access-5v8dh\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.149809 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149678 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbvg\" (UniqueName: \"kubernetes.io/projected/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-kube-api-access-kjbvg\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.149865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.149825 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7ps\" (UniqueName: \"kubernetes.io/projected/3da0ba8c-d364-4754-9069-47fe4a2017a6-kube-api-access-pm7ps\") pod \"volume-data-source-validator-58c9f5f46d-ppsxz\" (UID: \"3da0ba8c-d364-4754-9069-47fe4a2017a6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ppsxz" Oct 30 19:34:37.150286 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150261 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk4s5\" (UniqueName: \"kubernetes.io/projected/df601210-c5b1-4f3f-939b-bb8a11c3759d-kube-api-access-nk4s5\") pod \"kas-connection-checker-685895c7c9-kv78f\" (UID: \"df601210-c5b1-4f3f-939b-bb8a11c3759d\") " pod="kube-system/kas-connection-checker-685895c7c9-kv78f" Oct 30 19:34:37.150378 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150355 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:34:37.150427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150380 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:34:37.150537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150516 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvtz\" (UniqueName: \"kubernetes.io/projected/498b6873-9b05-4819-8d44-c6bbab466828-kube-api-access-qrvtz\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.150714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150692 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtr9\" (UniqueName: \"kubernetes.io/projected/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-kube-api-access-8mtr9\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.150826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.150885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150862 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.150962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150944 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s4x\" (UniqueName: \"kubernetes.io/projected/34d9f428-c41a-47ae-9ae5-9d93c76941ef-kube-api-access-z6s4x\") pod \"migrator-66df5d496b-sddsk\" (UID: \"34d9f428-c41a-47ae-9ae5-9d93c76941ef\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-sddsk" Oct 30 19:34:37.150962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.150956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwlv\" (UniqueName: \"kubernetes.io/projected/3fc71932-8adb-4e3a-83e8-acfe87c1c896-kube-api-access-xqwlv\") pod \"network-check-target-8f2fj\" (UID: \"3fc71932-8adb-4e3a-83e8-acfe87c1c896\") " pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:37.151169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.151155 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:34:37.152624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.152605 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.154792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.154767 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6c85830-f826-4df6-a48f-372c870e653a-ovnkube-script-lib\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.154859 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.154838 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.160019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.159998 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73261ca6-68ed-4f8d-bc9e-f4eb802b75a4-trusted-ca\") pod \"console-operator-7cd56b8dbc-zl8sn\" (UID: \"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:37.161221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.161205 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-frzlq\" (UID: \"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae\") " pod="openshift-insights/insights-operator-7c4986d449-frzlq" Oct 30 19:34:37.178182 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.178154 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:37.214595 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214544 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.214595 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214573 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.214595 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214589 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214604 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214637 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214661 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-cnibin\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214661 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214685 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-dbus\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214696 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b001525d-fb68-4e3f-b967-f1a8d43f620b-kubelet-config\") pod \"global-pull-secret-syncer-shwzj\" (UID: \"b001525d-fb68-4e3f-b967-f1a8d43f620b\") " pod="kube-system/global-pull-secret-syncer-shwzj" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214699 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214702 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214727 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-sys\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214728 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.214746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214750 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04a17b98-bb85-4bde-b1b4-29d1b39ad1c5-host\") pod \"node-ca-fr4xc\" (UID: \"04a17b98-bb85-4bde-b1b4-29d1b39ad1c5\") " pod="openshift-image-registry/node-ca-fr4xc" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214750 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214774 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214664 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysconfig\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214793 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214771 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-ovn\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.214827 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.71480828 +0000 UTC m=+1.844042197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214873 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-modprobe-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214880 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.214896 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.214911 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214928 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214906 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214945 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-netns\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.214961 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.714948723 +0000 UTC m=+1.844182679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.214986 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.214991 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215006 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215028 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215064 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-log-socket\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215071 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.715049983 +0000 UTC m=+1.844283902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215067 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-kubelet\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215105 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-registration-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215107 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215131 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215146 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215148 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-bin\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215171 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215189 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-hostroot\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215198 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.215203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215208 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-var-lib-kubelet\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215253 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215272 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-netns\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215304 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215327 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215349 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215365 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215367 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-crio-socket\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215376 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-conf-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215376 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-systemd\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215379 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-k8s-cni-cncf-io\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215413 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-multus\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215423 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215431 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-kubernetes\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215418 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-systemd-units\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215452 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215472 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215487 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215526 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215534 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-host\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215544 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-etc-kubernetes\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215550 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215522 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-run-multus-certs\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215559 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-systemd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215585 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215584 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215592 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-system-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215614 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215622 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.715612287 +0000 UTC m=+1.844846188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215650 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215667 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7cde5a04-1c75-44b3-8e49-bda52bb78768-hosts-file\") pod \"node-resolver-jw9s8\" (UID: \"7cde5a04-1c75-44b3-8e49-bda52bb78768\") " pod="openshift-dns/node-resolver-jw9s8" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215681 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215687 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215696 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215718 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215716 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-socket-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215736 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.715722403 +0000 UTC m=+1.844956324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215746 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-var-lib-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215752 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-cni-dir\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215753 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215762 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215787 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215795 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-run-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215814 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215832 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215841 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-node-log\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215851 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215868 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215882 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215881 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-slash\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215910 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215924 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-cni-netd\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215936 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215959 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215968 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-device-dir\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.215984 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.715974158 +0000 UTC m=+1.845208057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215935 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.215964 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e-host-slash\") pod \"iptables-alerter-przhx\" (UID: \"bdc913d7-1df1-409d-a0bd-9a13b9d4ab1e\") " pod="openshift-network-operator/iptables-alerter-przhx" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216002 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.715989813 +0000 UTC m=+1.845223712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216025 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-conf\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216038 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216057 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216072 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216086 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216097 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-system-cni-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216105 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216124 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216128 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-multus-socket-dir-parent\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216101 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-etc-openvswitch\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216138 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216165 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-cnibin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216170 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216175 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216187 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-etc-sysctl-d\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216139 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-lib-modules\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216191 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216224 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216197 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed54fa06-9be0-44dc-bf64-f48ea87b081f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7rbb\" (UID: \"ed54fa06-9be0-44dc-bf64-f48ea87b081f\") " pod="openshift-multus/multus-additional-cni-plugins-k7rbb" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216251 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216229 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216273 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216294 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.716285797 +0000 UTC m=+1.845519695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216300 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6c85830-f826-4df6-a48f-372c870e653a-host-kubelet\") pod \"ovnkube-node-pfvgl\" (UID: \"a6c85830-f826-4df6-a48f-372c870e653a\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216308 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216322 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9dcadcb7-1c64-4956-bf5c-489946be6487-run\") pod \"tuned-qzmns\" (UID: \"9dcadcb7-1c64-4956-bf5c-489946be6487\") " pod="openshift-cluster-node-tuning-operator/tuned-qzmns" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216325 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.216532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216359 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216385 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216396 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8699c88-8f03-4a7b-90f7-152a0118d517-host-var-lib-cni-bin\") pod \"multus-l5hwp\" (UID: \"e8699c88-8f03-4a7b-90f7-152a0118d517\") " pod="openshift-multus/multus-l5hwp" Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216364 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216416 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216431 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216454 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea3ab265-3e11-41d3-9192-e40c7588e1f5-sys-fs\") pod \"aws-ebs-csi-driver-node-qx2gv\" (UID: \"ea3ab265-3e11-41d3-9192-e40c7588e1f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx2gv" Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216463 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.716449185 +0000 UTC m=+1.845683101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.216487 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:37.716471264 +0000 UTC m=+1.845705185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:37.218850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.216705 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.311783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.311757 4959 scope.go:122] "RemoveContainer" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" Oct 30 19:34:37.326037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.326021 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:34:37.462285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.462259 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jw9s8_7cde5a04-1c75-44b3-8e49-bda52bb78768/dns-node-resolver/0.log" Oct 30 19:34:37.721886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.721810 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:37.721886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.721845 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:37.721886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.721872 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.721892 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721944 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721972 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721979 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.721986 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721997 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721975 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.721998 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.721983736 +0000 UTC m=+2.851217655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722062 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722066 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722052435 +0000 UTC m=+2.851286335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722080 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722074414 +0000 UTC m=+2.851308314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722092 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722087237 +0000 UTC m=+2.851321136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:37.722117 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722104 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722098706 +0000 UTC m=+2.851332605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.722141 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.722162 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.722180 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.722216 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:37.722246 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722271 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722260194 +0000 UTC m=+2.851494097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722287 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722305 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722313 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722322 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722331 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722316768 +0000 UTC m=+2.851550668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722351 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722338953 +0000 UTC m=+2.851572890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722365 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722358668 +0000 UTC m=+2.851592567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722306 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:37.722599 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:37.722387 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:38.722381688 +0000 UTC m=+2.851615638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:38.050395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.050320 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:34:38.050792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.050757 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/1.log" Oct 30 19:34:38.050828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.050790 4959 generic.go:356] "Generic (PLEG): container finished" podID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerID="2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43" exitCode=255 Oct 30 19:34:38.050934 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.050910 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" event={"ID":"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4","Type":"ContainerDied","Data":"2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43"} Oct 30 19:34:38.051008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.050969 4959 scope.go:122] "RemoveContainer" containerID="1e460e49e17e8938bb53121e7bf5ee7622132723eb892d8fc175e904eeca7edd" Oct 30 19:34:38.051320 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.051300 4959 scope.go:122] "RemoveContainer" containerID="2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43" Oct 30 19:34:38.051548 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.051519 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:38.258877 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.258852 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fr4xc_04a17b98-bb85-4bde-b1b4-29d1b39ad1c5/node-ca/0.log" Oct 30 19:34:38.734288 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734248 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:38.734288 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734289 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734307 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734334 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734417 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734432 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734420411 +0000 UTC m=+4.863654310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734448 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734417 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734478 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:38.734519 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734421 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734536 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734552 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734560 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734566 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734525 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734490011 +0000 UTC m=+4.863723932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : secret "router-metrics-certs-default" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734603 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734586999 +0000 UTC m=+4.863820912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734617 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.73461074 +0000 UTC m=+4.863844638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734629 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734623134 +0000 UTC m=+4.863857033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734639 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734634166 +0000 UTC m=+4.863868064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734677 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734706 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734738 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:38.734756 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734805 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734805 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734815 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734818 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734848 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls podName:eb7bc2f1-d85e-4656-b3a2-ca8c46710e20 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734835676 +0000 UTC m=+4.864069592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q8jlb" (UID: "eb7bc2f1-d85e-4656-b3a2-ca8c46710e20") : secret "insights-runtime-extractor-tls" not found Oct 30 19:34:38.734874 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734879 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734870287 +0000 UTC m=+4.864104186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:38.735413 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734895 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734886242 +0000 UTC m=+4.864120140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:38.735413 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:38.734908 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:40.734901735 +0000 UTC m=+4.864135634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:39.055229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:39.055145 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:34:39.258471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:39.258441 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r65qm_ae1ef3c9-11c5-48c2-b1db-166270b55b09/serve-healthcheck-canary/0.log" Oct 30 19:34:39.459072 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:39.459038 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-sddsk_34d9f428-c41a-47ae-9ae5-9d93c76941ef/migrator/0.log" Oct 30 19:34:39.859798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:39.859716 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5_51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374/kube-storage-version-migrator-operator/0.log" Oct 30 19:34:40.086139 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.086084 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.247270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.247235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.356716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.356687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.756308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756272 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:40.756308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756309 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756328 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756357 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756437 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756454 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756454 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756461 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle podName:498b6873-9b05-4819-8d44-c6bbab466828 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756444531 +0000 UTC m=+8.885678449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle") pod "router-default-fd9894b6f-skqbz" (UID: "498b6873-9b05-4819-8d44-c6bbab466828") : configmap references non-existent config key: service-ca.crt Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756458 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6775f6f89c-7npjm: secret "image-registry-tls" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756546 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756527355 +0000 UTC m=+8.885761271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756555 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756572 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls podName:ab02b8d2-0815-412a-a732-ef750ca1130b nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756557322 +0000 UTC m=+8.885791254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls") pod "image-registry-6775f6f89c-7npjm" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b") : secret "image-registry-tls" not found Oct 30 19:34:40.756594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756572 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-755b5c4488-4xx52: secret "image-registry-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756607 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756613 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls podName:b936af65-9699-44af-bbec-bd298c20d1fc nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756601196 +0000 UTC m=+8.885835098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls") pod "image-registry-755b5c4488-4xx52" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc") : secret "image-registry-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756672 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756682 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756702 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756707 4959 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756730 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756715829 +0000 UTC m=+8.885949729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756783 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756787 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756823 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.75681305 +0000 UTC m=+8.886046965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.756845 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756861 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756896 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert podName:5e27c42b-524e-4f97-b89b-87047a1d32f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756885514 +0000 UTC m=+8.886119413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mb2fw" (UID: "5e27c42b-524e-4f97-b89b-87047a1d32f7") : secret "networking-console-plugin-cert" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.756953 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:34:40.757045 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:40.757000 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls podName:3d7a9e09-ccd3-4c01-92fd-41844e62b60f nodeName:}" failed. No retries permitted until 2026-10-30 19:34:44.756988268 +0000 UTC m=+8.886222181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls") pod "dns-default-sg9nh" (UID: "3d7a9e09-ccd3-4c01-92fd-41844e62b60f") : secret "dns-default-metrics-tls" not found Oct 30 19:34:40.759028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.759006 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/498b6873-9b05-4819-8d44-c6bbab466828-metrics-certs\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:40.759028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.759022 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb7bc2f1-d85e-4656-b3a2-ca8c46710e20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q8jlb\" (UID: \"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20\") " pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:40.806179 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.806149 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.926670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:40.926632 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q8jlb" Oct 30 19:34:41.225404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.225361 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q8jlb"] Oct 30 19:34:41.229635 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:41.229607 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7bc2f1_d85e_4656_b3a2_ca8c46710e20.slice/crio-5e0b6054bcd6423ee4ebe5007111a7cab4baf939060f8134f6c686fd8693b66b WatchSource:0}: Error finding container 5e0b6054bcd6423ee4ebe5007111a7cab4baf939060f8134f6c686fd8693b66b: Status 404 returned error can't find the container with id 5e0b6054bcd6423ee4ebe5007111a7cab4baf939060f8134f6c686fd8693b66b Oct 30 19:34:41.278677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.278658 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:34:41.524284 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.524195 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:41.524654 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.524637 4959 scope.go:122] "RemoveContainer" containerID="2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43" Oct 30 19:34:41.524868 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:41.524848 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:41.558299 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.558264 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:41.949977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:41.949942 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.067147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.067066 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8jlb" event={"ID":"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20","Type":"ContainerStarted","Data":"cd26a1220d0191197f97faba2ba5f118b84c403ce1041b568b163f44c619b7c6"} Oct 30 19:34:42.067147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.067122 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8jlb" event={"ID":"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20","Type":"ContainerStarted","Data":"ef28e1ea97936d9ad047df27a5c60ae33d853df2c17eb393309ce4c1bf706dc2"} Oct 30 19:34:42.067147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.067141 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8jlb" event={"ID":"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20","Type":"ContainerStarted","Data":"5e0b6054bcd6423ee4ebe5007111a7cab4baf939060f8134f6c686fd8693b66b"} Oct 30 19:34:42.149951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.149915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.355331 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.355252 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.653839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.653807 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:42.762129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:42.762096 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.549472 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:43.549438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.662389 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:43.662355 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.950959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:43.950925 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.077019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.076985 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q8jlb" event={"ID":"eb7bc2f1-d85e-4656-b3a2-ca8c46710e20","Type":"ContainerStarted","Data":"f3aa5b9f8dc41ee4596918d8a58eae69dc91c3ee5d44272122ddfb8c2f95dd63"} Oct 30 19:34:44.147779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.147749 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.351545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.351450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.459674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.459648 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.753373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.753346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.795509 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795472 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:44.795646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795521 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:44.795646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795560 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:44.795646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795592 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:44.795646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795627 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:44.795851 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.795702 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:34:44.795851 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.795749 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:44.795851 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.795766 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls podName:cdbd83b4-3edb-4b0b-aed1-27ca01f92641 nodeName:}" failed. No retries permitted until 2026-10-30 19:34:52.795750123 +0000 UTC m=+16.924984026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-lbrlp" (UID: "cdbd83b4-3edb-4b0b-aed1-27ca01f92641") : secret "samples-operator-tls" not found Oct 30 19:34:44.795851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795786 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:44.795851 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.795816 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls podName:3e7e7bcf-5f36-44d1-9232-167eebe939aa nodeName:}" failed. No retries permitted until 2026-10-30 19:34:52.795797045 +0000 UTC m=+16.925030958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-5484c" (UID: "3e7e7bcf-5f36-44d1-9232-167eebe939aa") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:34:44.796111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795881 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:44.796111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.795913 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:44.796111 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.796037 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:34:44.796111 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:44.796076 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs podName:36088f70-cd5c-4911-ac09-cfb10529c20a nodeName:}" failed. No retries permitted until 2026-10-30 19:34:52.796062872 +0000 UTC m=+16.925296788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs") pod "network-metrics-daemon-57wdt" (UID: "36088f70-cd5c-4911-ac09-cfb10529c20a") : secret "metrics-daemon-secret" not found Oct 30 19:34:44.796323 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.796242 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/498b6873-9b05-4819-8d44-c6bbab466828-service-ca-bundle\") pod \"router-default-fd9894b6f-skqbz\" (UID: \"498b6873-9b05-4819-8d44-c6bbab466828\") " pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:44.798290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.798270 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d7a9e09-ccd3-4c01-92fd-41844e62b60f-metrics-tls\") pod \"dns-default-sg9nh\" (UID: \"3d7a9e09-ccd3-4c01-92fd-41844e62b60f\") " pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:44.798546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.798522 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"image-registry-6775f6f89c-7npjm\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:44.798691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.798672 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e27c42b-524e-4f97-b89b-87047a1d32f7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mb2fw\" (UID: \"5e27c42b-524e-4f97-b89b-87047a1d32f7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:44.798804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.798785 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"image-registry-755b5c4488-4xx52\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:44.811597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.811578 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:44.814123 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.814102 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:44.814257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.814242 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" Oct 30 19:34:44.824364 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.824239 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:44.825317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.825299 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:44.938291 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.938263 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:44.938578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.938562 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:44.993262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:44.993230 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.145160 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.145130 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-fd9894b6f-skqbz"] Oct 30 19:34:45.148712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.148691 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sg9nh"] Oct 30 19:34:45.149095 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:45.149072 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498b6873_9b05_4819_8d44_c6bbab466828.slice/crio-bdcf745f6a78f614d43db0e6d77fd563f7586f4bce082d65b353875f517a8700 WatchSource:0}: Error finding container bdcf745f6a78f614d43db0e6d77fd563f7586f4bce082d65b353875f517a8700: Status 404 returned error can't find the container with id bdcf745f6a78f614d43db0e6d77fd563f7586f4bce082d65b353875f517a8700 Oct 30 19:34:45.152189 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:45.152171 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7a9e09_ccd3_4c01_92fd_41844e62b60f.slice/crio-542dded661834526b7768dc663f5710b751bf657d6d9fb723159e1c9ca76edf9 WatchSource:0}: Error finding container 542dded661834526b7768dc663f5710b751bf657d6d9fb723159e1c9ca76edf9: Status 404 returned error can't find the container with id 542dded661834526b7768dc663f5710b751bf657d6d9fb723159e1c9ca76edf9 Oct 30 19:34:45.156429 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.156411 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:34:45.156830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.156797 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mb2fw"] Oct 30 19:34:45.160638 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:45.160615 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27c42b_524e_4f97_b89b_87047a1d32f7.slice/crio-04beeb79b107cfe541c8e6708aa6929ff58485123f8546031b22c156cd6341a6 WatchSource:0}: Error finding container 04beeb79b107cfe541c8e6708aa6929ff58485123f8546031b22c156cd6341a6: Status 404 returned error can't find the container with id 04beeb79b107cfe541c8e6708aa6929ff58485123f8546031b22c156cd6341a6 Oct 30 19:34:45.161435 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:45.161412 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab02b8d2_0815_412a_a732_ef750ca1130b.slice/crio-7a4372cf263a674bbfe48d457909c8eb46d040864c3a071144d464962060fac6 WatchSource:0}: Error finding container 7a4372cf263a674bbfe48d457909c8eb46d040864c3a071144d464962060fac6: Status 404 returned error can't find the container with id 7a4372cf263a674bbfe48d457909c8eb46d040864c3a071144d464962060fac6 Oct 30 19:34:45.164385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.164371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:34:45.168733 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:45.168711 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb936af65_9699_44af_bbec_bd298c20d1fc.slice/crio-6806eda30073f7ddc7c5f500156c1edc0fb208274d368311f17ba09266030809 WatchSource:0}: Error finding container 6806eda30073f7ddc7c5f500156c1edc0fb208274d368311f17ba09266030809: Status 404 returned error can't find the container with id 6806eda30073f7ddc7c5f500156c1edc0fb208274d368311f17ba09266030809 Oct 30 19:34:45.550204 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.550128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.859167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.859077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.967691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:45.967656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.070908 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:46.070871 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:34:46.086481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.086442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" event={"ID":"ab02b8d2-0815-412a-a732-ef750ca1130b","Type":"ContainerStarted","Data":"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099"} Oct 30 19:34:46.086674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.086538 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" event={"ID":"ab02b8d2-0815-412a-a732-ef750ca1130b","Type":"ContainerStarted","Data":"7a4372cf263a674bbfe48d457909c8eb46d040864c3a071144d464962060fac6"} Oct 30 19:34:46.089084 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.089052 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" event={"ID":"b936af65-9699-44af-bbec-bd298c20d1fc","Type":"ContainerStarted","Data":"929ee82f8e272a45306192e096e51980a2ffd1c1b892f4e273fbb03cfdf65b0e"} Oct 30 19:34:46.089219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.089099 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" event={"ID":"b936af65-9699-44af-bbec-bd298c20d1fc","Type":"ContainerStarted","Data":"6806eda30073f7ddc7c5f500156c1edc0fb208274d368311f17ba09266030809"} Oct 30 19:34:46.091982 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.091954 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fd9894b6f-skqbz" event={"ID":"498b6873-9b05-4819-8d44-c6bbab466828","Type":"ContainerStarted","Data":"941ce6a1ec080c12e6fe52f276b93ea9f593244caae9674b4759bc86a2423043"} Oct 30 19:34:46.092115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.092002 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-fd9894b6f-skqbz" event={"ID":"498b6873-9b05-4819-8d44-c6bbab466828","Type":"ContainerStarted","Data":"bdcf745f6a78f614d43db0e6d77fd563f7586f4bce082d65b353875f517a8700"} Oct 30 19:34:46.093679 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.093650 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg9nh" event={"ID":"3d7a9e09-ccd3-4c01-92fd-41844e62b60f","Type":"ContainerStarted","Data":"542dded661834526b7768dc663f5710b751bf657d6d9fb723159e1c9ca76edf9"} Oct 30 19:34:46.095347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.095316 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" event={"ID":"5e27c42b-524e-4f97-b89b-87047a1d32f7","Type":"ContainerStarted","Data":"04beeb79b107cfe541c8e6708aa6929ff58485123f8546031b22c156cd6341a6"} Oct 30 19:34:46.146356 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.146327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.341819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.341784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.551920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.551880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.562745 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.562714 4959 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:46.563157 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.563142 4959 scope.go:122] "RemoveContainer" containerID="2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43" Oct 30 19:34:46.563357 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:34:46.563337 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zl8sn_openshift-console-operator(73261ca6-68ed-4f8d-bc9e-f4eb802b75a4)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" Oct 30 19:34:46.812353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.812329 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:46.815096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:46.815073 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:47.100253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.100155 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mb2fw" event={"ID":"5e27c42b-524e-4f97-b89b-87047a1d32f7","Type":"ContainerStarted","Data":"a15e94cf727295acfbaef0d300f126788978c61cae249b71678ba6778b4556c9"} Oct 30 19:34:47.101879 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.101855 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg9nh" event={"ID":"3d7a9e09-ccd3-4c01-92fd-41844e62b60f","Type":"ContainerStarted","Data":"09cda5e7bff6c1e14c23f98f8b70956e32c242e2e2bd3343e14b49403ca18d56"} Oct 30 19:34:47.102030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.101893 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sg9nh" event={"ID":"3d7a9e09-ccd3-4c01-92fd-41844e62b60f","Type":"ContainerStarted","Data":"54b92d2b7388d1572526e255597b7b71007f5d73681e833d3ffeb5ad3960ce11"} Oct 30 19:34:47.102030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.101917 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:47.102030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.102004 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:47.102205 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.102152 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:47.757872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.757839 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:47.948598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.948564 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:47.989592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:47.989562 4959 scope.go:122] "RemoveContainer" containerID="2e8d9d06b7541d37ef60a4b2cbeb34f31a4a8806bdb7abc90c6ca809fc24ec43" Oct 30 19:34:48.107613 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.107590 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:34:48.107989 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.107722 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" event={"ID":"73261ca6-68ed-4f8d-bc9e-f4eb802b75a4","Type":"ContainerStarted","Data":"39d68c688513d3474dfe9689bfd55baa808ec2bdef1c9a9fd8c79f2c61e4d596"} Oct 30 19:34:48.108049 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.108029 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:48.108268 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.108252 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:48.108959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.108932 4959 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zl8sn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Oct 30 19:34:48.109020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.108975 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Oct 30 19:34:48.147940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.147909 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.354758 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.354669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.551017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.550986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.716546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.716484 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:34:48.755532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.755488 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.860423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.860392 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:48.963006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:48.962974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.065765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.065696 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.114573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.114546 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" Oct 30 19:34:49.281314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.281283 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.488442 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.488412 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.595188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.595157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.751066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.750989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.803684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.803658 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:49.803823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.803787 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:34:49.805053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.805034 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-fd9894b6f-skqbz" Oct 30 19:34:49.942395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:49.942367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.147511 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:50.147478 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.349608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:50.349569 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.569625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:50.569541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.803771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:50.803726 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:34:50.954399 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:50.954367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.166862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:51.166830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.277644 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:51.277561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.642234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:51.642196 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.946564 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:51.946457 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bzkf" Oct 30 19:34:52.865367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.865334 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:52.865913 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.865384 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:52.865913 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.865419 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:52.867956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.867932 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e7e7bcf-5f36-44d1-9232-167eebe939aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-5484c\" (UID: \"3e7e7bcf-5f36-44d1-9232-167eebe939aa\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:52.868308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.868291 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36088f70-cd5c-4911-ac09-cfb10529c20a-metrics-certs\") pod \"network-metrics-daemon-57wdt\" (UID: \"36088f70-cd5c-4911-ac09-cfb10529c20a\") " pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:52.868394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.868375 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdbd83b4-3edb-4b0b-aed1-27ca01f92641-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-lbrlp\" (UID: \"cdbd83b4-3edb-4b0b-aed1-27ca01f92641\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:52.912774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.912740 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" Oct 30 19:34:52.912927 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.912800 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" Oct 30 19:34:52.923943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:52.923920 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57wdt" Oct 30 19:34:53.250621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:53.250580 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57wdt"] Oct 30 19:34:53.254431 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:53.254404 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36088f70_cd5c_4911_ac09_cfb10529c20a.slice/crio-fd61861a3dc16fca1d81d11067cb1ce37783b796b9b3e0413beb29621e9223e0 WatchSource:0}: Error finding container fd61861a3dc16fca1d81d11067cb1ce37783b796b9b3e0413beb29621e9223e0: Status 404 returned error can't find the container with id fd61861a3dc16fca1d81d11067cb1ce37783b796b9b3e0413beb29621e9223e0 Oct 30 19:34:53.261511 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:53.261468 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-5484c"] Oct 30 19:34:53.262075 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:53.262053 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp"] Oct 30 19:34:53.267388 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:34:53.267363 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7e7bcf_5f36_44d1_9232_167eebe939aa.slice/crio-0b5860fa4930c3dabf0d6d44b4c63eab53889d7ce400fa0096bec3d695c9ee24 WatchSource:0}: Error finding container 0b5860fa4930c3dabf0d6d44b4c63eab53889d7ce400fa0096bec3d695c9ee24: Status 404 returned error can't find the container with id 0b5860fa4930c3dabf0d6d44b4c63eab53889d7ce400fa0096bec3d695c9ee24 Oct 30 19:34:54.039232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.039173 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfvgl" Oct 30 19:34:54.058142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.057281 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:34:54.120666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.120604 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:54.138745 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.138660 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" event={"ID":"cdbd83b4-3edb-4b0b-aed1-27ca01f92641","Type":"ContainerStarted","Data":"302d852329c462a841474b82ce305c8a550c7b447bd901d4fb0f7d1b01f1e46d"} Oct 30 19:34:54.140941 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.140860 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57wdt" event={"ID":"36088f70-cd5c-4911-ac09-cfb10529c20a","Type":"ContainerStarted","Data":"fd61861a3dc16fca1d81d11067cb1ce37783b796b9b3e0413beb29621e9223e0"} Oct 30 19:34:54.143481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.143427 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" event={"ID":"3e7e7bcf-5f36-44d1-9232-167eebe939aa","Type":"ContainerStarted","Data":"0b5860fa4930c3dabf0d6d44b4c63eab53889d7ce400fa0096bec3d695c9ee24"} Oct 30 19:34:54.326648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.326566 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:54.434935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:54.434873 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.153551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:55.153522 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57wdt" event={"ID":"36088f70-cd5c-4911-ac09-cfb10529c20a","Type":"ContainerStarted","Data":"f22452efca75cb2f5f309e68c8cf09e3079d9317dd85f1a55b83dc80a3476f40"} Oct 30 19:34:55.153864 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:55.153575 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57wdt" event={"ID":"36088f70-cd5c-4911-ac09-cfb10529c20a","Type":"ContainerStarted","Data":"e1599210a6d30228ab3241aa31dbd35604e07bec4f6380af1b60bfc8143a9743"} Oct 30 19:34:55.803826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:55.803783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:55.918999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:55.918957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:56.166260 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:56.166223 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-5484c" event={"ID":"3e7e7bcf-5f36-44d1-9232-167eebe939aa","Type":"ContainerStarted","Data":"f6d18f028fc203bf28f8c30b46352fc6b517d82dbc45b3c5c940cbf7bdac4340"} Oct 30 19:34:56.168569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:56.168535 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" event={"ID":"cdbd83b4-3edb-4b0b-aed1-27ca01f92641","Type":"ContainerStarted","Data":"bc40748820f04f2fe07c22d83fa7fcd9dfc1b6eb56876f98860f70c7b2e9cfdb"} Oct 30 19:34:56.168714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:56.168592 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-lbrlp" event={"ID":"cdbd83b4-3edb-4b0b-aed1-27ca01f92641","Type":"ContainerStarted","Data":"1bb94da5f703552050923c63066c21705092c49d27494a2e7b3f81d31e5d1adf"} Oct 30 19:34:56.216253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:56.216210 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:56.353301 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:56.353262 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:57.110675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:57.110647 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sg9nh" Oct 30 19:34:57.226277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:57.226250 4959 patch_prober.go:29] interesting pod/image-registry-6775f6f89c-7npjm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:34:57.226645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:57.226293 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:34:59.339726 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:59.339694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.457160 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:59.457129 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.561071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:34:59.561027 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.808471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:00.808435 4959 patch_prober.go:29] interesting pod/image-registry-6775f6f89c-7npjm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:35:00.808937 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:00.808503 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:35:02.288112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:02.288080 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:03.853006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:03.852972 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.063754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.063718 4959 patch_prober.go:29] interesting pod/image-registry-755b5c4488-4xx52 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:35:04.063934 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.063774 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:35:04.064109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.064074 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.204765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.204722 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.205470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.205452 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nc5gg"] Oct 30 19:35:04.240254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.240227 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.248985 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-root\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249019 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-metrics-client-ca\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249049 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-tls\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249172 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlddt\" (UniqueName: \"kubernetes.io/projected/ff46f19b-f529-49cb-a027-c1b0d28295e1-kube-api-access-jlddt\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249197 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-sys\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249226 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-textfile\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249321 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-wtmp\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249377 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249354 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.249417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.249378 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.268295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.268268 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:35:04.268447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.268386 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:35:04.268447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.268428 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:35:04.268557 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.268443 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:35:04.268557 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.268463 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dpldx\"" Oct 30 19:35:04.319817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.319782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.350032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.349992 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-tls\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350051 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlddt\" (UniqueName: \"kubernetes.io/projected/ff46f19b-f529-49cb-a027-c1b0d28295e1-kube-api-access-jlddt\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350081 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-sys\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350122 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-textfile\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350156 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-wtmp\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350180 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-sys\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350212 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-root\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350315 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-wtmp\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350322 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-metrics-client-ca\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350437 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff46f19b-f529-49cb-a027-c1b0d28295e1-root\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.350669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350593 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-textfile\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.351135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.350896 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-metrics-client-ca\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.351135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.351014 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-accelerators-collector-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.352712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.352692 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.352878 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.352859 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff46f19b-f529-49cb-a027-c1b0d28295e1-node-exporter-tls\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.370993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.370957 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlddt\" (UniqueName: \"kubernetes.io/projected/ff46f19b-f529-49cb-a027-c1b0d28295e1-kube-api-access-jlddt\") pod \"node-exporter-nc5gg\" (UID: \"ff46f19b-f529-49cb-a027-c1b0d28295e1\") " pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.543438 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.543355 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.591465 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.591432 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nc5gg" Oct 30 19:35:04.634568 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:35:04.634537 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff46f19b_f529_49cb_a027_c1b0d28295e1.slice/crio-4a8e676d1fa3f0805d922b36cbf38dd12f4d6dd3f0796d17ce9a49f5600aedad WatchSource:0}: Error finding container 4a8e676d1fa3f0805d922b36cbf38dd12f4d6dd3f0796d17ce9a49f5600aedad: Status 404 returned error can't find the container with id 4a8e676d1fa3f0805d922b36cbf38dd12f4d6dd3f0796d17ce9a49f5600aedad Oct 30 19:35:04.648174 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.648149 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.756633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.756604 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.967693 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:04.967656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.082447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:05.082403 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.205390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:05.205358 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nc5gg" event={"ID":"ff46f19b-f529-49cb-a027-c1b0d28295e1","Type":"ContainerStarted","Data":"4a8e676d1fa3f0805d922b36cbf38dd12f4d6dd3f0796d17ce9a49f5600aedad"} Oct 30 19:35:05.264338 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:05.264256 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.436354 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:05.436311 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.543233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:05.543148 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:06.211689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:06.211593 4959 generic.go:356] "Generic (PLEG): container finished" podID="ff46f19b-f529-49cb-a027-c1b0d28295e1" containerID="2b4d4c150dbabd2b388be9ac289f1da983332b2ca33ed985cd88e7a811714017" exitCode=0 Oct 30 19:35:06.212150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:06.211670 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nc5gg" event={"ID":"ff46f19b-f529-49cb-a027-c1b0d28295e1","Type":"ContainerDied","Data":"2b4d4c150dbabd2b388be9ac289f1da983332b2ca33ed985cd88e7a811714017"} Oct 30 19:35:07.165321 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.165287 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:07.218240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.218209 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nc5gg" event={"ID":"ff46f19b-f529-49cb-a027-c1b0d28295e1","Type":"ContainerStarted","Data":"315cbb38bcb6470103f3378c0fa0bdf0dcf3a014cf63a421abf3da6e95ddc119"} Oct 30 19:35:07.218746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.218259 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nc5gg" event={"ID":"ff46f19b-f529-49cb-a027-c1b0d28295e1","Type":"ContainerStarted","Data":"e5ae63942da4f2515d7d1c758dded088ee9c1fc338d8589b93f37266c8161096"} Oct 30 19:35:07.264105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.264053 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nc5gg" podStartSLOduration=1.873542499 podStartE2EDuration="3.264039236s" totalImagesPullingTime="1.295474193s" totalInitContainerRuntime="95.022544ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:07.263079166 +0000 UTC m=+31.392313086" watchObservedRunningTime="2026-10-30 19:35:07.264039236 +0000 UTC m=+31.393273158" Oct 30 19:35:07.270150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.270124 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:07.446959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.446887 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:07.554181 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:07.554149 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.053960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:08.053927 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8f2fj" Oct 30 19:35:08.144202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:08.144174 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.249520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:08.249468 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.998534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:08.998472 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:09.228033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:09.228000 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.808163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:10.808138 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:35:11.106298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:11.106213 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.225978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:11.225936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.061691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:14.061666 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:35:14.932295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:14.932263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.045182 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:15.045147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.064527 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:35:16.064471 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:35:17.850765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:17.850734 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:35:18.021298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:18.021261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:18.132905 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:18.132872 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:18.652571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:18.652529 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:18.756996 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:18.756947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.091136 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.091054 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" containerName="registry" containerID="cri-o://929ee82f8e272a45306192e096e51980a2ffd1c1b892f4e273fbb03cfdf65b0e" gracePeriod=30 Oct 30 19:35:19.183480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.183450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.267074 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.267040 4959 generic.go:356] "Generic (PLEG): container finished" podID="b936af65-9699-44af-bbec-bd298c20d1fc" containerID="929ee82f8e272a45306192e096e51980a2ffd1c1b892f4e273fbb03cfdf65b0e" exitCode=0 Oct 30 19:35:19.267283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.267104 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" event={"ID":"b936af65-9699-44af-bbec-bd298c20d1fc","Type":"ContainerDied","Data":"929ee82f8e272a45306192e096e51980a2ffd1c1b892f4e273fbb03cfdf65b0e"} Oct 30 19:35:19.295202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.295170 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.331033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.331007 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:35:19.366829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366741 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.366829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366788 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366837 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366897 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.366988 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.367014 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367299 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.367051 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") pod \"b936af65-9699-44af-bbec-bd298c20d1fc\" (UID: \"b936af65-9699-44af-bbec-bd298c20d1fc\") " Oct 30 19:35:19.367438 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.367410 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:19.367648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.367623 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:19.369723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.369686 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:19.369853 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.369735 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:19.369908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.369886 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "kube-api-access-9mc5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:19.369908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.369892 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:19.369973 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.369919 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:19.373762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.373742 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted" pod "b936af65-9699-44af-bbec-bd298c20d1fc" (UID: "b936af65-9699-44af-bbec-bd298c20d1fc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:19.467634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467602 4959 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-installation-pull-secrets\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467629 4959 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b936af65-9699-44af-bbec-bd298c20d1fc-ca-trust-extracted\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467639 4959 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-registry-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467647 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mc5x\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-kube-api-access-9mc5x\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467658 4959 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b936af65-9699-44af-bbec-bd298c20d1fc-image-registry-private-configuration\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467667 4959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-trusted-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467675 4959 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b936af65-9699-44af-bbec-bd298c20d1fc-bound-sa-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.467943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.467684 4959 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b936af65-9699-44af-bbec-bd298c20d1fc-registry-certificates\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:19.528906 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.528874 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:19.637723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:19.637692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:20.275347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:20.275243 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" event={"ID":"b936af65-9699-44af-bbec-bd298c20d1fc","Type":"ContainerDied","Data":"6806eda30073f7ddc7c5f500156c1edc0fb208274d368311f17ba09266030809"} Oct 30 19:35:20.275347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:20.275295 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-755b5c4488-4xx52" Oct 30 19:35:20.275347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:20.275344 4959 scope.go:122] "RemoveContainer" containerID="929ee82f8e272a45306192e096e51980a2ffd1c1b892f4e273fbb03cfdf65b0e" Oct 30 19:35:20.337081 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:20.337043 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:35:20.348201 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:20.348166 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-755b5c4488-4xx52"] Oct 30 19:35:21.994361 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:21.994330 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" path="/var/lib/kubelet/pods/b936af65-9699-44af-bbec-bd298c20d1fc/volumes" Oct 30 19:35:23.639567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:23.639536 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.760380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:23.760337 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.226841 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:26.226811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.373648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:26.373616 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:26.485870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:26.485768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.582548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:27.582488 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:27.696213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:27.696162 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:29.801801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:29.801767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:29.925092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:29.925065 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.058805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:30.058734 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.163057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:30.163023 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:34.542295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:34.542262 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:34.666722 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:34.666692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:42.871892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:42.871834 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" containerID="cri-o://2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099" gracePeriod=30 Oct 30 19:35:43.049932 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.049903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:43.116670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.116645 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:35:43.262406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262371 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262416 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262460 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262480 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262520 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262552 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262589 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.262627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.262611 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") pod \"ab02b8d2-0815-412a-a732-ef750ca1130b\" (UID: \"ab02b8d2-0815-412a-a732-ef750ca1130b\") " Oct 30 19:35:43.263433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.263075 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:43.263433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.263110 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:43.265662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.265626 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:43.265771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.265659 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:43.265771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.265694 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:43.265771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.265701 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:43.265930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.265907 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "kube-api-access-mp845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:43.269730 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.269708 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted" pod "ab02b8d2-0815-412a-a732-ef750ca1130b" (UID: "ab02b8d2-0815-412a-a732-ef750ca1130b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:43.364918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364883 4959 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.364918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364920 4959 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-bound-sa-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364933 4959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-trusted-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364947 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp845\" (UniqueName: \"kubernetes.io/projected/ab02b8d2-0815-412a-a732-ef750ca1130b-kube-api-access-mp845\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364983 4959 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ab02b8d2-0815-412a-a732-ef750ca1130b-registry-certificates\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.364999 4959 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-image-registry-private-configuration\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.365015 4959 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ab02b8d2-0815-412a-a732-ef750ca1130b-ca-trust-extracted\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.365162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.365028 4959 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ab02b8d2-0815-412a-a732-ef750ca1130b-installation-pull-secrets\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:35:43.369477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.369447 4959 generic.go:356] "Generic (PLEG): container finished" podID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerID="2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099" exitCode=0 Oct 30 19:35:43.369642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.369515 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" Oct 30 19:35:43.369642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.369538 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" event={"ID":"ab02b8d2-0815-412a-a732-ef750ca1130b","Type":"ContainerDied","Data":"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099"} Oct 30 19:35:43.369642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.369611 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6775f6f89c-7npjm" event={"ID":"ab02b8d2-0815-412a-a732-ef750ca1130b","Type":"ContainerDied","Data":"7a4372cf263a674bbfe48d457909c8eb46d040864c3a071144d464962060fac6"} Oct 30 19:35:43.369642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.369630 4959 scope.go:122] "RemoveContainer" containerID="2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099" Oct 30 19:35:43.378963 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.378945 4959 scope.go:122] "RemoveContainer" containerID="2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099" Oct 30 19:35:43.379232 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:35:43.379211 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099\": container with ID starting with 2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099 not found: ID does not exist" containerID="2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099" Oct 30 19:35:43.379296 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.379240 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099"} err="failed to get container status \"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099\": rpc error: code = NotFound desc = could not find container \"2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099\": container with ID starting with 2d249f3cf330987b836ee94700c23612f050c109431386c933018e87c5a77099 not found: ID does not exist" Oct 30 19:35:43.416855 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.416830 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:35:43.423132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.423112 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6775f6f89c-7npjm"] Oct 30 19:35:43.993940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:43.993904 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" path="/var/lib/kubelet/pods/ab02b8d2-0815-412a-a732-ef750ca1130b/volumes" Oct 30 19:35:44.065680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:44.065647 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.173300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:44.173263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.064847 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:35:46.064808 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:35:46.458811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:46.458774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.568103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:46.568071 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.767211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:46.767132 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:46.883598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:46.883566 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.541470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:49.541440 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.647042 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:49.647009 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.017952 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:58.017919 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.131380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:35:58.131346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.116874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:00.116838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.224076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:00.224033 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.348966 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:00.348938 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.493945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:00.493911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.597843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:00.597813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.440253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:01.440217 4959 generic.go:356] "Generic (PLEG): container finished" podID="51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374" containerID="44febf9f6dc5a8b18f587d55d7c49b800f8a58abe3ce7f3849948176c85eac43" exitCode=0 Oct 30 19:36:01.440664 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:01.440290 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" event={"ID":"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374","Type":"ContainerDied","Data":"44febf9f6dc5a8b18f587d55d7c49b800f8a58abe3ce7f3849948176c85eac43"} Oct 30 19:36:01.440664 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:01.440609 4959 scope.go:122] "RemoveContainer" containerID="44febf9f6dc5a8b18f587d55d7c49b800f8a58abe3ce7f3849948176c85eac43" Oct 30 19:36:02.446123 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:02.446086 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-rcsn5" event={"ID":"51f9c455-a5a6-4c80-8bcd-cd0fe9d4d374","Type":"ContainerStarted","Data":"7c4d720dfd7ad423c1c3b1703d2b0cc99fb73754d80acafb00723229f77f5698"} Oct 30 19:36:02.447576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:02.447554 4959 generic.go:356] "Generic (PLEG): container finished" podID="410e9153-45cc-485e-ba92-efa60cca301e" containerID="d685ed919c5d7bba51c1c66f2608858a9c1fe6b4e050a4e876bded70a75a37c4" exitCode=0 Oct 30 19:36:02.447696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:02.447621 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" event={"ID":"410e9153-45cc-485e-ba92-efa60cca301e","Type":"ContainerDied","Data":"d685ed919c5d7bba51c1c66f2608858a9c1fe6b4e050a4e876bded70a75a37c4"} Oct 30 19:36:02.447890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:02.447878 4959 scope.go:122] "RemoveContainer" containerID="d685ed919c5d7bba51c1c66f2608858a9c1fe6b4e050a4e876bded70a75a37c4" Oct 30 19:36:03.452461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:03.452426 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-65x2c" event={"ID":"410e9153-45cc-485e-ba92-efa60cca301e","Type":"ContainerStarted","Data":"3c70995a9c66f5c38b564acd79c53b751a30e632af6e39fd70297c5035a3c15e"} Oct 30 19:36:04.531762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:04.531719 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:04.639485 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:04.639442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:05.464197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:05.464166 4959 generic.go:356] "Generic (PLEG): container finished" podID="58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae" containerID="ba16583277c4ab7778acb70aead594f5d71b9f8575720c2c649948ebf92bbfc5" exitCode=0 Oct 30 19:36:05.464434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:05.464236 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-frzlq" event={"ID":"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae","Type":"ContainerDied","Data":"ba16583277c4ab7778acb70aead594f5d71b9f8575720c2c649948ebf92bbfc5"} Oct 30 19:36:05.464603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:05.464586 4959 scope.go:122] "RemoveContainer" containerID="ba16583277c4ab7778acb70aead594f5d71b9f8575720c2c649948ebf92bbfc5" Oct 30 19:36:06.469659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:06.469627 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-frzlq" event={"ID":"58fa447b-1c5e-4eab-aeda-1c0f6d80f1ae","Type":"ContainerStarted","Data":"37512d7013c86b5b9a2aaf2345931b45a67ea35fa9cedd6545f7a73c51a34bd0"} Oct 30 19:36:16.063165 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:36:16.063124 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:36:26.364236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:26.364205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:41.774878 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:36:41.774839 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:46.064139 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:36:46.064096 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:37:16.065203 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:37:16.065164 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:37:26.478677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:37:26.478637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.747058 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:37:33.747018 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.868465 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:37:33.868435 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:46.065645 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:37:46.065608 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:38:06.327169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:06.327139 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:06.434935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:06.434898 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:14.995383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:14.995345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.100037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:15.100003 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:16.064614 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:38:16.064573 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:38:17.895094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:17.895058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:18.011623 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:18.011591 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:26.582029 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:26.581997 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.543084 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:34.543047 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.671139 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:34.671101 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:34.925918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:34.925882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:46.064853 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:38:46.064817 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:38:51.125487 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:38:51.125455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:16.064963 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:39:16.064926 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:39:26.725098 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:39:26.725059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:36.004125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:39:36.004098 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:39:36.009794 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:39:36.009765 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:39:46.064327 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:39:46.064281 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:40:16.065246 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:40:16.065209 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:40:26.858933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:26.858897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.668929 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:29.668893 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.796958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:29.796929 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.922369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:29.922276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.145372 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145342 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145669 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" containerName="registry" Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145681 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" containerName="registry" Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145691 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145696 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145762 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b936af65-9699-44af-bbec-bd298c20d1fc" containerName="registry" Oct 30 19:40:31.145774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.145771 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab02b8d2-0815-412a-a732-ef750ca1130b" containerName="registry" Oct 30 19:40:31.148620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.148605 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.164597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164566 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.164863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164835 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:40:31.164863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164837 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:40:31.164863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164858 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-f48d6\"" Oct 30 19:40:31.164863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164840 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:40:31.165061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.164868 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:40:31.178551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.178529 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:40:31.221408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221361 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221464 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221568 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221613 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221667 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221685 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmb4p\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.221781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.221705 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.285823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.285784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.322512 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322465 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322525 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322661 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322640 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322703 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322772 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322729 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322772 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322748 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmb4p\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.322772 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.322767 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.323407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.323380 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.325113 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.325074 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.325236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.325140 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.325236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.325141 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.325236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.325210 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.350569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.350529 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.350724 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.350708 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmb4p\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\") pod \"istiod-openshift-gateway-7876bfcb74-676sg\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.460040 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.459950 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:31.804743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.804671 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:40:31.808091 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:40:31.808066 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667615b9_e88d_45e9_b387_65b3cbc75c1c.slice/crio-5320cf59a3fc95f250301fadcc5857df75688b9fdd88f146e2d2a3a46d6aca43 WatchSource:0}: Error finding container 5320cf59a3fc95f250301fadcc5857df75688b9fdd88f146e2d2a3a46d6aca43: Status 404 returned error can't find the container with id 5320cf59a3fc95f250301fadcc5857df75688b9fdd88f146e2d2a3a46d6aca43 Oct 30 19:40:31.810033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:31.810016 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:40:32.412044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:32.412009 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" event={"ID":"667615b9-e88d-45e9-b387-65b3cbc75c1c","Type":"ContainerStarted","Data":"5320cf59a3fc95f250301fadcc5857df75688b9fdd88f146e2d2a3a46d6aca43"} Oct 30 19:40:32.473448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:32.473418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:35.031869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.031832 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:40:35.032181 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.031903 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:40:35.426172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.426138 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" event={"ID":"667615b9-e88d-45e9-b387-65b3cbc75c1c","Type":"ContainerStarted","Data":"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f"} Oct 30 19:40:35.426333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.426201 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:35.498143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.498073 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" podStartSLOduration=1.277304008 podStartE2EDuration="4.4980513s" totalImagesPullingTime="3.220747292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:35.483697414 +0000 UTC m=+359.612931336" watchObservedRunningTime="2026-10-30 19:40:35.4980513 +0000 UTC m=+359.627285224" Oct 30 19:40:35.768884 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.768795 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:35.877901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:35.877848 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.431485 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:36.431457 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:40:36.618144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:36.618113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.767033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:36.766996 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.878658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:36.878624 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.056865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:37.056787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:40.313797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:40:40.313769 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.064563 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:40:46.064526 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:41:08.378610 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:08.378575 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.488174 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:08.488143 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:14.124346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:14.124316 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.064051 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:41:16.064015 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:41:18.369072 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:18.369039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.498609 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:18.498574 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.727175 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:18.727141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.524411 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.524385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.635562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.635516 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.869563 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.869460 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb"] Oct 30 19:41:19.872788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.872772 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.893545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.893516 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb"] Oct 30 19:41:19.914785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.914756 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgtg\" (UniqueName: \"kubernetes.io/projected/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-kube-api-access-9xgtg\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.914910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.914788 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.914910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.914805 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.915132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.914940 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.915132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.914962 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.915132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.915054 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:19.947935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.947910 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:41:19.948053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.947938 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:41:19.948148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.948135 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:41:19.948699 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.948688 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:41:19.948743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.948692 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:41:19.948785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:19.948740 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zx96v\"" Oct 30 19:41:20.015549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015525 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.015697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015553 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.015697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015588 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.015697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015614 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgtg\" (UniqueName: \"kubernetes.io/projected/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-kube-api-access-9xgtg\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.015870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015794 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.015870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.015830 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.016146 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.016117 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-manifests\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.018095 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.018069 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-tmp\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.018203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.018181 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-webhook-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.018257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.018222 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-metrics-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.018423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.018398 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.019229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.019210 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.043305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.043283 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgtg\" (UniqueName: \"kubernetes.io/projected/6e372fa2-9f21-40aa-8a02-cc3699e73a2d-kube-api-access-9xgtg\") pod \"opendatahub-operator-controller-manager-54879d4f6b-cs4mb\" (UID: \"6e372fa2-9f21-40aa-8a02-cc3699e73a2d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.124616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.124554 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.187418 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.187397 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:20.253755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.253705 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.368886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.368857 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.528422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.528372 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.597524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.595620 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb"] Oct 30 19:41:20.658604 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.658538 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.776674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:20.776642 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:21.593900 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:21.593867 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" event={"ID":"6e372fa2-9f21-40aa-8a02-cc3699e73a2d","Type":"ContainerStarted","Data":"0190cb4db48b9751ed987cd663f221a517bfd7e3f239efc9740287e9aba7c97e"} Oct 30 19:41:22.773413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:22.773367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.355053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:23.355021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.602760 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:23.602674 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" event={"ID":"6e372fa2-9f21-40aa-8a02-cc3699e73a2d","Type":"ContainerStarted","Data":"d5ee02a598a2ad5954f2de0f8509bc174a2a609235217a1df32e93ae595ad8b1"} Oct 30 19:41:24.336894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:24.336850 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.608972 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:24.608880 4959 generic.go:356] "Generic (PLEG): container finished" podID="6e372fa2-9f21-40aa-8a02-cc3699e73a2d" containerID="d5ee02a598a2ad5954f2de0f8509bc174a2a609235217a1df32e93ae595ad8b1" exitCode=0 Oct 30 19:41:24.608972 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:24.608936 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" event={"ID":"6e372fa2-9f21-40aa-8a02-cc3699e73a2d","Type":"ContainerDied","Data":"d5ee02a598a2ad5954f2de0f8509bc174a2a609235217a1df32e93ae595ad8b1"} Oct 30 19:41:25.480785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:25.480744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:25.614719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:25.614678 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" event={"ID":"6e372fa2-9f21-40aa-8a02-cc3699e73a2d","Type":"ContainerStarted","Data":"f07d0bb15a8f2a46846b4ae62e03b7a36ac012a09125b049654a4ac28ca6a7a0"} Oct 30 19:41:25.614916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:25.614839 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:25.711131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:25.711073 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" podStartSLOduration=3.200965847 podStartE2EDuration="6.711057135s" totalImagesPullingTime="3.033997224s" totalInitContainerRuntime="476.094064ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:25.710545541 +0000 UTC m=+409.839779459" watchObservedRunningTime="2026-10-30 19:41:25.711057135 +0000 UTC m=+409.840291050" Oct 30 19:41:25.885892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:25.885863 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.014422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:26.014359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.988483 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:26.988449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.043931 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:28.043903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.520963 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:28.520916 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.099944 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:29.099908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.402474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:29.402437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:30.139018 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:30.138985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.425902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:34.425865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.569662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:34.569602 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.621390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:36.621355 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54879d4f6b-cs4mb" Oct 30 19:41:36.648449 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:36.648421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.804862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:36.804819 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.130368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:38.130330 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:38.305620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:38.305579 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.177235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:39.177199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.453798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:39.453708 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.344964 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:43.344934 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.463738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:43.463694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.744939 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:43.744902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.868719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:43.868686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:44.684316 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:44.684246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:44.863704 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:44.863653 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:46.064636 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:41:46.064599 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:41:46.450164 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:46.450132 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:47.151088 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:47.151057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:47.272866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:47.272834 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.788674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:48.788645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.905456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:48.905425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.943754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:50.943715 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.075766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:51.075728 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.207718 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.207682 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.359798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.359757 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.673460 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.673429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.943466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.943385 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz"] Oct 30 19:41:52.946615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.946598 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:52.973844 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.973816 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:41:52.981583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.981562 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:41:52.981708 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.981566 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vcfxs\"" Oct 30 19:41:52.983923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.983901 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz"] Oct 30 19:41:52.992012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:52.991976 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.089381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.089346 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77496d52-d5c3-4e02-88f6-2036d4547a8c-tmp\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.089646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.089415 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77496d52-d5c3-4e02-88f6-2036d4547a8c-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.089646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.089560 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96cz\" (UniqueName: \"kubernetes.io/projected/77496d52-d5c3-4e02-88f6-2036d4547a8c-kube-api-access-w96cz\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.146222 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.146187 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.190863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.190833 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77496d52-d5c3-4e02-88f6-2036d4547a8c-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.191060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.190884 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96cz\" (UniqueName: \"kubernetes.io/projected/77496d52-d5c3-4e02-88f6-2036d4547a8c-kube-api-access-w96cz\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.191060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.190919 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77496d52-d5c3-4e02-88f6-2036d4547a8c-tmp\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.193565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.193471 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77496d52-d5c3-4e02-88f6-2036d4547a8c-tmp\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.193695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.193663 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77496d52-d5c3-4e02-88f6-2036d4547a8c-tls-certs\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.222436 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.222398 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96cz\" (UniqueName: \"kubernetes.io/projected/77496d52-d5c3-4e02-88f6-2036d4547a8c-kube-api-access-w96cz\") pod \"kube-auth-proxy-66b985bdbd-tc2wz\" (UID: \"77496d52-d5c3-4e02-88f6-2036d4547a8c\") " pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.259469 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.259434 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" Oct 30 19:41:53.276995 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.276959 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.399650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.399622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.560974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.560948 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz"] Oct 30 19:41:53.562827 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:41:53.562799 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77496d52_d5c3_4e02_88f6_2036d4547a8c.slice/crio-7591121c147401a7f74f53d57611495dacdcdf64d07b28b50a37c7b87900d019 WatchSource:0}: Error finding container 7591121c147401a7f74f53d57611495dacdcdf64d07b28b50a37c7b87900d019: Status 404 returned error can't find the container with id 7591121c147401a7f74f53d57611495dacdcdf64d07b28b50a37c7b87900d019 Oct 30 19:41:53.731470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:53.731418 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" event={"ID":"77496d52-d5c3-4e02-88f6-2036d4547a8c","Type":"ContainerStarted","Data":"7591121c147401a7f74f53d57611495dacdcdf64d07b28b50a37c7b87900d019"} Oct 30 19:41:54.126006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:54.125961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.236898 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:54.236868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.716779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:54.716733 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.025245 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.025164 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:41:55.031106 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.030304 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:41:55.053537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.053487 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:41:55.054768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.054744 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:41:55.065020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.064995 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:41:55.080396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.080368 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhvjj\"" Oct 30 19:41:55.105514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.105457 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mdw\" (UniqueName: \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\") pod \"authorino-56c6c9ddd5-k5w6s\" (UID: \"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d\") " pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:41:55.206794 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.206761 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mdw\" (UniqueName: \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\") pod \"authorino-56c6c9ddd5-k5w6s\" (UID: \"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d\") " pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:41:55.230339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.230301 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mdw\" (UniqueName: \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\") pod \"authorino-56c6c9ddd5-k5w6s\" (UID: \"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d\") " pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:41:55.360351 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.360274 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:41:55.665789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.665766 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:41:55.667465 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:41:55.667436 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1e94fb4_65a9_4408_b807_27c9b5cf8c5d.slice/crio-094133bdc496ed57300a0014453767ef09676d6f3e268958277a0666dc0db7c9 WatchSource:0}: Error finding container 094133bdc496ed57300a0014453767ef09676d6f3e268958277a0666dc0db7c9: Status 404 returned error can't find the container with id 094133bdc496ed57300a0014453767ef09676d6f3e268958277a0666dc0db7c9 Oct 30 19:41:55.741890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.741851 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" event={"ID":"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d","Type":"ContainerStarted","Data":"094133bdc496ed57300a0014453767ef09676d6f3e268958277a0666dc0db7c9"} Oct 30 19:41:55.828243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:55.828209 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.749707 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:56.749667 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" event={"ID":"77496d52-d5c3-4e02-88f6-2036d4547a8c","Type":"ContainerStarted","Data":"02e6935276b6ad27fd96b7808835f98b7693cd8aa48967579ec852d53c93126d"} Oct 30 19:41:56.794143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:56.794076 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66b985bdbd-tc2wz" podStartSLOduration=2.26659287 podStartE2EDuration="4.794054755s" totalImagesPullingTime="2.527461885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:56.793427145 +0000 UTC m=+440.922661066" watchObservedRunningTime="2026-10-30 19:41:56.794054755 +0000 UTC m=+440.923288673" Oct 30 19:41:57.602406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:57.602357 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.745832 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:57.745800 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.759822 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:58.759785 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" event={"ID":"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d","Type":"ContainerStarted","Data":"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151"} Oct 30 19:41:58.803976 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:58.803925 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" podStartSLOduration=2.272752628 podStartE2EDuration="4.803909297s" totalImagesPullingTime="2.531156669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:58.803249754 +0000 UTC m=+442.932483676" watchObservedRunningTime="2026-10-30 19:41:58.803909297 +0000 UTC m=+442.933143248" Oct 30 19:41:59.500757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:59.500716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.605791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:41:59.605760 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.221959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:00.221921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.000780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:03.000751 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.118578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:03.118548 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.609813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:03.609781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.725650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:03.725615 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.130460 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:07.130427 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.279935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:07.279903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.374271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:11.374223 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.484378 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:11.484351 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.975079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:11.975053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.096325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:12.096294 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.569564 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:12.569521 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.678991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:12.678959 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:14.923624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:14.923590 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.072565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:15.072526 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.745094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:15.745049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:16.064236 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:42:16.064159 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:42:25.181428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:25.181397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:27.133864 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:27.133828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.057587 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:35.057554 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.175208 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:35.175172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.291233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:35.291202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.408801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:35.408773 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.010804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:39.010771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.146065 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:39.146026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:40.611960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:40.611910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:40.748149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:40.748117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:46.064298 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:42:46.064250 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:42:57.684473 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.684443 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9"] Oct 30 19:42:57.687553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.687536 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.719070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.719046 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9"] Oct 30 19:42:57.794974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.794942 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.794979 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj96x\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-kube-api-access-gj96x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.795086 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.795131 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.795200 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a13b5e-fa09-42e7-93fd-54108c02ac43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.795245 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.795277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.795265 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896361 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896404 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896439 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a13b5e-fa09-42e7-93fd-54108c02ac43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896468 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896649 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896708 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.896754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.896741 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj96x\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-kube-api-access-gj96x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.897454 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.897427 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.899044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.899017 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.899193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.899172 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.899476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.899462 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a13b5e-fa09-42e7-93fd-54108c02ac43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.899656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.899640 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.928149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.928126 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj96x\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-kube-api-access-gj96x\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.928368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.928351 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a13b5e-fa09-42e7-93fd-54108c02ac43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vcqz9\" (UID: \"11a13b5e-fa09-42e7-93fd-54108c02ac43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:57.998060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:57.997990 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:58.336344 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.336317 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9"] Oct 30 19:42:58.338815 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:42:58.338789 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a13b5e_fa09_42e7_93fd_54108c02ac43.slice/crio-4de9759d8376a70842a31050bb98bf8012a29f79dbdfbd3bece44a7da27184fe WatchSource:0}: Error finding container 4de9759d8376a70842a31050bb98bf8012a29f79dbdfbd3bece44a7da27184fe: Status 404 returned error can't find the container with id 4de9759d8376a70842a31050bb98bf8012a29f79dbdfbd3bece44a7da27184fe Oct 30 19:42:58.340774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.340746 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:42:58.340841 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.340803 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:42:58.504871 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.504806 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.632952 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.632912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.745650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.745624 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.879722 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:58.879671 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.004457 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.004377 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" event={"ID":"11a13b5e-fa09-42e7-93fd-54108c02ac43","Type":"ContainerStarted","Data":"1a48aed3df1b7f795f8e133ca19bcbca59d02524950f4c95cbd0b372ef8ea12d"} Oct 30 19:42:59.004457 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.004438 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" event={"ID":"11a13b5e-fa09-42e7-93fd-54108c02ac43","Type":"ContainerStarted","Data":"4de9759d8376a70842a31050bb98bf8012a29f79dbdfbd3bece44a7da27184fe"} Oct 30 19:42:59.004752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.004600 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:42:59.006468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.006418 4959 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vcqz9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:42:59.006606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.006542 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" podUID="11a13b5e-fa09-42e7-93fd-54108c02ac43" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:42:59.023616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.023584 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.375960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:42:59.375868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.008318 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.008294 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" Oct 30 19:43:00.094619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.094569 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vcqz9" podStartSLOduration=3.094554042 podStartE2EDuration="3.094554042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:59.055816074 +0000 UTC m=+503.185049998" watchObservedRunningTime="2026-10-30 19:43:00.094554042 +0000 UTC m=+504.223787964" Oct 30 19:43:00.141309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.141279 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:43:00.141516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.141472 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" podUID="667615b9-e88d-45e9-b387-65b3cbc75c1c" containerName="discovery" containerID="cri-o://490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f" gracePeriod=30 Oct 30 19:43:00.381680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.381654 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:43:00.515902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.515872 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.515902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.515905 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.515935 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.515957 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.515979 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.516032 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.516093 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\") pod \"667615b9-e88d-45e9-b387-65b3cbc75c1c\" (UID: \"667615b9-e88d-45e9-b387-65b3cbc75c1c\") " Oct 30 19:43:00.516409 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.516343 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:00.518401 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518366 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:00.518543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518406 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "kube-api-access-jmb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:00.518791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518766 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:00.518860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518816 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:00.518915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518897 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:00.518915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.518899 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token" pod "667615b9-e88d-45e9-b387-65b3cbc75c1c" (UID: "667615b9-e88d-45e9-b387-65b3cbc75c1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:00.617167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617096 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617125 4959 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/667615b9-e88d-45e9-b387-65b3cbc75c1c-local-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617137 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmb4p\" (UniqueName: \"kubernetes.io/projected/667615b9-e88d-45e9-b387-65b3cbc75c1c-kube-api-access-jmb4p\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617149 4959 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-ca-configmap\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617161 4959 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-kubeconfig\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617174 4959 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-cacerts\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:00.617410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:00.617186 4959 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/667615b9-e88d-45e9-b387-65b3cbc75c1c-istio-csr-dns-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:01.013106 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.013072 4959 generic.go:356] "Generic (PLEG): container finished" podID="667615b9-e88d-45e9-b387-65b3cbc75c1c" containerID="490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f" exitCode=0 Oct 30 19:43:01.013598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.013123 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" Oct 30 19:43:01.013598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.013168 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" event={"ID":"667615b9-e88d-45e9-b387-65b3cbc75c1c","Type":"ContainerDied","Data":"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f"} Oct 30 19:43:01.013598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.013227 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg" event={"ID":"667615b9-e88d-45e9-b387-65b3cbc75c1c","Type":"ContainerDied","Data":"5320cf59a3fc95f250301fadcc5857df75688b9fdd88f146e2d2a3a46d6aca43"} Oct 30 19:43:01.013598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.013258 4959 scope.go:122] "RemoveContainer" containerID="490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f" Oct 30 19:43:01.022374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.022360 4959 scope.go:122] "RemoveContainer" containerID="490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f" Oct 30 19:43:01.022654 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:43:01.022637 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f\": container with ID starting with 490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f not found: ID does not exist" containerID="490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f" Oct 30 19:43:01.022725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.022662 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f"} err="failed to get container status \"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f\": rpc error: code = NotFound desc = could not find container \"490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f\": container with ID starting with 490149902bd6093fc05ce7c47daf99cc7c720c49e8d1e3f4d35242d538df390f not found: ID does not exist" Oct 30 19:43:01.080963 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.080938 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:43:01.085221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.085203 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-676sg"] Oct 30 19:43:01.539057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.539030 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.653674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.653646 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.771922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.771893 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.898917 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.898889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.993996 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:01.993970 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667615b9-e88d-45e9-b387-65b3cbc75c1c" path="/var/lib/kubelet/pods/667615b9-e88d-45e9-b387-65b3cbc75c1c/volumes" Oct 30 19:43:02.036209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:02.036181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.631893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:03.631857 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.747798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:03.747769 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:09.116602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:09.116561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:09.239797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:09.239753 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.630856 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.630809 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.755448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.755418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.869690 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.869658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.880917 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.880853 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:10.881206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.881193 4959 cpu_manager.go:414] "containerMap: removing container" podUID="667615b9-e88d-45e9-b387-65b3cbc75c1c" containerName="discovery" Oct 30 19:43:10.881255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.881206 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667615b9-e88d-45e9-b387-65b3cbc75c1c" containerName="discovery" Oct 30 19:43:10.881290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.881259 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="667615b9-e88d-45e9-b387-65b3cbc75c1c" containerName="discovery" Oct 30 19:43:10.885209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.885194 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.890376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.890348 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsd8s\" (UniqueName: \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.890515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.890432 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.890515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.890460 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.890609 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.890560 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.914530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.914461 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:43:10.914689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.914528 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcrrd\"" Oct 30 19:43:10.914689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.914555 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:43:10.914689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.914646 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:43:10.914846 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.914797 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:43:10.925076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.925051 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:10.981785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.981758 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.990795 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.990768 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsd8s\" (UniqueName: \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.990901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.990805 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.990966 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.990949 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.991051 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.991023 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.991678 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.991651 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.993435 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.993410 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:10.993574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:10.993469 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:11.012233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:11.012214 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsd8s\" (UniqueName: \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\") pod \"maas-api-6d669c7fcd-fw7g5\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:11.197168 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:11.197150 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:11.505349 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:43:11.505321 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b3bc5f_7cbd_4dae_bfc3_25a73c0efa14.slice/crio-6549850388fda34ad6432aefeff8580be2d57da8578d55b97377528406a8aa41 WatchSource:0}: Error finding container 6549850388fda34ad6432aefeff8580be2d57da8578d55b97377528406a8aa41: Status 404 returned error can't find the container with id 6549850388fda34ad6432aefeff8580be2d57da8578d55b97377528406a8aa41 Oct 30 19:43:11.505522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:11.505436 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:12.056302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:12.056256 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" event={"ID":"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14","Type":"ContainerStarted","Data":"6549850388fda34ad6432aefeff8580be2d57da8578d55b97377528406a8aa41"} Oct 30 19:43:13.995485 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:13.995454 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:14.070981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:14.070946 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" event={"ID":"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14","Type":"ContainerStarted","Data":"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e"} Oct 30 19:43:14.071820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:14.071776 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:14.121048 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:14.120954 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" podStartSLOduration=1.780185441 podStartE2EDuration="4.120940003s" totalImagesPullingTime="2.340754562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:14.120225576 +0000 UTC m=+518.249459497" watchObservedRunningTime="2026-10-30 19:43:14.120940003 +0000 UTC m=+518.250173927" Oct 30 19:43:14.123815 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:14.123763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:15.866283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:15.866251 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:15.976811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:15.976784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:16.063385 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:43:16.063354 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:43:17.854596 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.854565 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh"] Oct 30 19:43:17.857714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.857697 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:17.875316 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.875290 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 19:43:17.888737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.888711 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 19:43:17.888737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.888715 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-ff2md\"" Oct 30 19:43:17.888919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.888774 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 19:43:17.888919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.888782 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:43:17.889897 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.889867 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh"] Oct 30 19:43:17.939709 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.939682 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c567f\" (UniqueName: \"kubernetes.io/projected/a3771588-95d8-475b-9c96-91c6b6d8a8ff-kube-api-access-c567f\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:17.939830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.939720 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-cert\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:17.939830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.939744 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3771588-95d8-475b-9c96-91c6b6d8a8ff-manager-config\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:17.939908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.939888 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-metrics-certs\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:17.975421 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:17.975395 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:43:18.040342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.040319 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-metrics-certs\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.040480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.040411 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c567f\" (UniqueName: \"kubernetes.io/projected/a3771588-95d8-475b-9c96-91c6b6d8a8ff-kube-api-access-c567f\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.040480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.040447 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-cert\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.040645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.040524 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3771588-95d8-475b-9c96-91c6b6d8a8ff-manager-config\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.041192 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.041171 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a3771588-95d8-475b-9c96-91c6b6d8a8ff-manager-config\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.042967 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.042947 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-metrics-certs\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.043173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.043154 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3771588-95d8-475b-9c96-91c6b6d8a8ff-cert\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.062156 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.062135 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c567f\" (UniqueName: \"kubernetes.io/projected/a3771588-95d8-475b-9c96-91c6b6d8a8ff-kube-api-access-c567f\") pod \"jobset-controller-manager-5fdf59b7-zzrlh\" (UID: \"a3771588-95d8-475b-9c96-91c6b6d8a8ff\") " pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.169126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.169104 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:18.467078 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:18.467050 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh"] Oct 30 19:43:18.467589 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:43:18.467566 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3771588_95d8_475b_9c96_91c6b6d8a8ff.slice/crio-55c46cff3e50128f19b205a0c55ad3684da0db9f1c50404ed448f0a752fc0afa WatchSource:0}: Error finding container 55c46cff3e50128f19b205a0c55ad3684da0db9f1c50404ed448f0a752fc0afa: Status 404 returned error can't find the container with id 55c46cff3e50128f19b205a0c55ad3684da0db9f1c50404ed448f0a752fc0afa Oct 30 19:43:19.091289 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:19.091260 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" event={"ID":"a3771588-95d8-475b-9c96-91c6b6d8a8ff","Type":"ContainerStarted","Data":"55c46cff3e50128f19b205a0c55ad3684da0db9f1c50404ed448f0a752fc0afa"} Oct 30 19:43:20.082227 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:20.082191 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:20.550845 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:20.550811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:20.762108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:20.762075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:22.108352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:22.108315 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" event={"ID":"a3771588-95d8-475b-9c96-91c6b6d8a8ff","Type":"ContainerStarted","Data":"1e92611b6ea054cec3430d9440bb6c93e4fcbf85676aca6ffcf5de54388e3300"} Oct 30 19:43:22.108756 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:22.108386 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:22.159188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:22.159143 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" podStartSLOduration=2.103164025 podStartE2EDuration="5.159130534s" totalImagesPullingTime="3.055966509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:22.157004154 +0000 UTC m=+526.286238076" watchObservedRunningTime="2026-10-30 19:43:22.159130534 +0000 UTC m=+526.288364528" Oct 30 19:43:23.372063 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.372032 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:43:23.372601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.372221 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" podUID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" containerName="authorino" containerID="cri-o://c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151" gracePeriod=30 Oct 30 19:43:23.618230 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.618206 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:43:23.685542 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.685513 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\" (UniqueName: \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\") pod \"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d\" (UID: \"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d\") " Oct 30 19:43:23.687813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.687788 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw" pod "f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" (UID: "f1e94fb4-65a9-4408-b807-27c9b5cf8c5d"). InnerVolumeSpecName "kube-api-access-79mdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:23.786824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:23.786799 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79mdw\" (UniqueName: \"kubernetes.io/projected/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d-kube-api-access-79mdw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:24.117412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.117378 4959 generic.go:356] "Generic (PLEG): container finished" podID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" containerID="c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151" exitCode=0 Oct 30 19:43:24.117569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.117420 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" Oct 30 19:43:24.117569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.117427 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" event={"ID":"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d","Type":"ContainerDied","Data":"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151"} Oct 30 19:43:24.117569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.117470 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-k5w6s" event={"ID":"f1e94fb4-65a9-4408-b807-27c9b5cf8c5d","Type":"ContainerDied","Data":"094133bdc496ed57300a0014453767ef09676d6f3e268958277a0666dc0db7c9"} Oct 30 19:43:24.117569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.117487 4959 scope.go:122] "RemoveContainer" containerID="c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151" Oct 30 19:43:24.125633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.125616 4959 scope.go:122] "RemoveContainer" containerID="c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151" Oct 30 19:43:24.125923 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:43:24.125901 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151\": container with ID starting with c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151 not found: ID does not exist" containerID="c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151" Oct 30 19:43:24.126019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.125929 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151"} err="failed to get container status \"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151\": rpc error: code = NotFound desc = could not find container \"c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151\": container with ID starting with c21c64edd61923c49c45d43717951cd9468cf054fe4b46b61883386e6c5c8151 not found: ID does not exist" Oct 30 19:43:24.170922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.170898 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:43:24.177858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:24.177837 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-k5w6s"] Oct 30 19:43:25.993557 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:25.993525 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" path="/var/lib/kubelet/pods/f1e94fb4-65a9-4408-b807-27c9b5cf8c5d/volumes" Oct 30 19:43:27.279020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:27.278981 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.749148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:30.749119 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:30.749620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:30.749298 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" podUID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" containerName="maas-api" containerID="cri-o://3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e" gracePeriod=30 Oct 30 19:43:30.990290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:30.990268 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:31.036999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.036916 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\") pod \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " Oct 30 19:43:31.036999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.036966 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\") pod \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " Oct 30 19:43:31.036999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.036993 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\") pod \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " Oct 30 19:43:31.037271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.037068 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\" (UniqueName: \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\") pod \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\" (UID: \"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14\") " Oct 30 19:43:31.039342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.039293 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca" pod "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" (UID: "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:31.039590 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.039562 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls" pod "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" (UID: "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:31.039710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.039641 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs" pod "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" (UID: "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:31.039835 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.039812 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s" pod "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" (UID: "d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14"). InnerVolumeSpecName "kube-api-access-jsd8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:31.138324 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.138284 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsd8s\" (UniqueName: \"kubernetes.io/projected/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-kube-api-access-jsd8s\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:31.138324 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.138319 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:31.138324 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.138333 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:31.138601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.138346 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:43:31.147076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.147048 4959 generic.go:356] "Generic (PLEG): container finished" podID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" containerID="3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e" exitCode=0 Oct 30 19:43:31.147237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.147110 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" Oct 30 19:43:31.147237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.147133 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" event={"ID":"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14","Type":"ContainerDied","Data":"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e"} Oct 30 19:43:31.147237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.147193 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5" event={"ID":"d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14","Type":"ContainerDied","Data":"6549850388fda34ad6432aefeff8580be2d57da8578d55b97377528406a8aa41"} Oct 30 19:43:31.147237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.147226 4959 scope.go:122] "RemoveContainer" containerID="3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e" Oct 30 19:43:31.156253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.156237 4959 scope.go:122] "RemoveContainer" containerID="3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e" Oct 30 19:43:31.156555 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:43:31.156529 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e\": container with ID starting with 3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e not found: ID does not exist" containerID="3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e" Oct 30 19:43:31.156611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.156569 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e"} err="failed to get container status \"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e\": rpc error: code = NotFound desc = could not find container \"3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e\": container with ID starting with 3f4f359c88f9ab7e2d9128a576c0ea6bfb74467e2f1190b9c8d4807c01126c9e not found: ID does not exist" Oct 30 19:43:31.201851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.201825 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:31.208669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.208650 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fw7g5"] Oct 30 19:43:31.993575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:31.993546 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" path="/var/lib/kubelet/pods/d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14/volumes" Oct 30 19:43:33.118115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:33.118089 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5fdf59b7-zzrlh" Oct 30 19:43:45.359283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:45.359249 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.479156 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:45.479125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:46.063917 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:43:46.063885 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:43:50.458008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:43:50.457971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.065250 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:44:16.065144 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:44:27.438624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:27.438577 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.029716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:36.029684 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:44:36.033081 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:36.033062 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:44:38.407795 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:38.407751 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:38.535357 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:38.535321 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.065548 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:44:46.065484 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:44:55.624778 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:55.624744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.712768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:57.712731 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.720477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.720444 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4"] Oct 30 19:44:59.720941 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.720922 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" containerName="maas-api" Oct 30 19:44:59.720991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.720945 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" containerName="maas-api" Oct 30 19:44:59.720991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.720971 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" containerName="authorino" Oct 30 19:44:59.720991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.720978 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" containerName="authorino" Oct 30 19:44:59.721107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.721070 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b3bc5f-7cbd-4dae-bfc3-25a73c0efa14" containerName="maas-api" Oct 30 19:44:59.721107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.721083 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e94fb4-65a9-4408-b807-27c9b5cf8c5d" containerName="authorino" Oct 30 19:44:59.724702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.724678 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.741280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.741239 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:44:59.741280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.741260 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:44:59.754657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.754610 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-674xl\"" Oct 30 19:44:59.754657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.754633 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:44:59.756020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.755993 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4"] Oct 30 19:44:59.822701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.822670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.840850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.840821 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.841023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.840862 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.841023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.840890 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.841023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.840946 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pf7\" (UniqueName: \"kubernetes.io/projected/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-kube-api-access-x5pf7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.841132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.841038 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.841132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.841061 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942418 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942668 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942581 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942668 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942621 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pf7\" (UniqueName: \"kubernetes.io/projected/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-kube-api-access-x5pf7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942675 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942711 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942773 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.942940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.942791 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.943032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.943012 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.943118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.943099 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.945047 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.945028 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.945184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.945167 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:44:59.967346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:44:59.967319 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pf7\" (UniqueName: \"kubernetes.io/projected/10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde-kube-api-access-x5pf7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5npp4\" (UID: \"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:00.040412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.040334 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:00.161054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.161017 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:00.164400 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.164383 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.189158 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.189138 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 19:45:00.189540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.189523 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:00.189701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.189685 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:00.195284 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.195256 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:00.244879 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.244847 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgkr\" (UniqueName: \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.245056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.244907 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.344722 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.344696 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4"] Oct 30 19:45:00.345724 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.345704 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgkr\" (UniqueName: \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.345782 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.345746 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.346329 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.346305 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.347119 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:00.347100 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a0dd10_7ce2_48e4_8fe6_e7c1b9019fde.slice/crio-d7f48f77139096ce3ef344d312766cd326af5372febdc101d8f89fcc08860902 WatchSource:0}: Error finding container d7f48f77139096ce3ef344d312766cd326af5372febdc101d8f89fcc08860902: Status 404 returned error can't find the container with id d7f48f77139096ce3ef344d312766cd326af5372febdc101d8f89fcc08860902 Oct 30 19:45:00.373707 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.373682 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgkr\" (UniqueName: \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\") pod \"maas-api-key-cleanup-29889825-hpbwp\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.477238 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.477207 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:00.488749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.488720 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" event={"ID":"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde","Type":"ContainerStarted","Data":"d7f48f77139096ce3ef344d312766cd326af5372febdc101d8f89fcc08860902"} Oct 30 19:45:00.786307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:00.786263 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:00.787733 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:00.787700 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014b7614_80f6_46bb_b643_b142e1b1b773.slice/crio-aa534c04c63172ca951c95aa8ef9b09491fb0596588e56cf96b9258af357e6bb WatchSource:0}: Error finding container aa534c04c63172ca951c95aa8ef9b09491fb0596588e56cf96b9258af357e6bb: Status 404 returned error can't find the container with id aa534c04c63172ca951c95aa8ef9b09491fb0596588e56cf96b9258af357e6bb Oct 30 19:45:01.217716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:01.217681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:01.495718 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:01.495595 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerStarted","Data":"aa534c04c63172ca951c95aa8ef9b09491fb0596588e56cf96b9258af357e6bb"} Oct 30 19:45:02.502019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:02.501989 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" event={"ID":"10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde","Type":"ContainerStarted","Data":"7cf5084106147e34641b0ee372c30c273d41c1ab5c5a14ae36a35ace50e89a12"} Oct 30 19:45:02.565637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:02.564652 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" podStartSLOduration=2.334661752 podStartE2EDuration="3.564633538s" totalImagesPullingTime="1.229971786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.563073038 +0000 UTC m=+626.692306949" watchObservedRunningTime="2026-10-30 19:45:02.564633538 +0000 UTC m=+626.693867461" Oct 30 19:45:02.637040 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:02.637002 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:03.041063 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:03.041028 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:03.507536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:03.507486 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerStarted","Data":"bdcfe766631a15f7ffc03c49276a843d544ecf185c00e2299baa280d0a4a07bc"} Oct 30 19:45:03.564772 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:03.564716 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" podStartSLOduration=1.136184044 podStartE2EDuration="3.564698606s" totalImagesPullingTime="2.428514562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:03.563875981 +0000 UTC m=+627.693109902" watchObservedRunningTime="2026-10-30 19:45:03.564698606 +0000 UTC m=+627.693932528" Oct 30 19:45:04.082956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.082921 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65"] Oct 30 19:45:04.086283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.086261 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.117753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.117724 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 19:45:04.119182 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.119159 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65"] Oct 30 19:45:04.177739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177702 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62ds\" (UniqueName: \"kubernetes.io/projected/890d2c73-01dc-453a-b0d6-9db8142c9980-kube-api-access-p62ds\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.177739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177744 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.177962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177778 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.177962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177951 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.178040 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177973 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890d2c73-01dc-453a-b0d6-9db8142c9980-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.178040 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.177995 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.202021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.201992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.279350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279262 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279304 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62ds\" (UniqueName: \"kubernetes.io/projected/890d2c73-01dc-453a-b0d6-9db8142c9980-kube-api-access-p62ds\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279330 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279361 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279405 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279420 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890d2c73-01dc-453a-b0d6-9db8142c9980-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279879 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279929 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279883 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.279929 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.279905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.282068 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.282044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890d2c73-01dc-453a-b0d6-9db8142c9980-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.282575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.282552 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890d2c73-01dc-453a-b0d6-9db8142c9980-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.313210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.313179 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62ds\" (UniqueName: \"kubernetes.io/projected/890d2c73-01dc-453a-b0d6-9db8142c9980-kube-api-access-p62ds\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65\" (UID: \"890d2c73-01dc-453a-b0d6-9db8142c9980\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.399909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.399874 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:04.513343 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.513311 4959 generic.go:356] "Generic (PLEG): container finished" podID="014b7614-80f6-46bb-b643-b142e1b1b773" containerID="bdcfe766631a15f7ffc03c49276a843d544ecf185c00e2299baa280d0a4a07bc" exitCode=60 Oct 30 19:45:04.513766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.513406 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerDied","Data":"bdcfe766631a15f7ffc03c49276a843d544ecf185c00e2299baa280d0a4a07bc"} Oct 30 19:45:04.513766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.513638 4959 scope.go:122] "RemoveContainer" containerID="bdcfe766631a15f7ffc03c49276a843d544ecf185c00e2299baa280d0a4a07bc" Oct 30 19:45:04.707996 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:04.707971 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65"] Oct 30 19:45:04.709203 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:04.709176 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890d2c73_01dc_453a_b0d6_9db8142c9980.slice/crio-a0957b144d17bdc2de7fd69455d205c0ff815a5b970b581b9e3671ee177e858c WatchSource:0}: Error finding container a0957b144d17bdc2de7fd69455d205c0ff815a5b970b581b9e3671ee177e858c: Status 404 returned error can't find the container with id a0957b144d17bdc2de7fd69455d205c0ff815a5b970b581b9e3671ee177e858c Oct 30 19:45:05.519808 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.519704 4959 generic.go:356] "Generic (PLEG): container finished" podID="014b7614-80f6-46bb-b643-b142e1b1b773" containerID="e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c" exitCode=60 Oct 30 19:45:05.519808 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.519796 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerDied","Data":"e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c"} Oct 30 19:45:05.520339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.519866 4959 scope.go:122] "RemoveContainer" containerID="bdcfe766631a15f7ffc03c49276a843d544ecf185c00e2299baa280d0a4a07bc" Oct 30 19:45:05.520339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.520104 4959 scope.go:122] "RemoveContainer" containerID="e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c" Oct 30 19:45:05.520445 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:05.520338 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-hpbwp_odh-ai-gateway-infra(014b7614-80f6-46bb-b643-b142e1b1b773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" Oct 30 19:45:05.521559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.521537 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" event={"ID":"890d2c73-01dc-453a-b0d6-9db8142c9980","Type":"ContainerStarted","Data":"5cd69661a47a12c635dcfaf29c8b2234f82ed6031767107756c808ec0cf5bd7d"} Oct 30 19:45:05.521631 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.521584 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" event={"ID":"890d2c73-01dc-453a-b0d6-9db8142c9980","Type":"ContainerStarted","Data":"a0957b144d17bdc2de7fd69455d205c0ff815a5b970b581b9e3671ee177e858c"} Oct 30 19:45:05.614596 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:05.614544 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" podStartSLOduration=1.435561414 podStartE2EDuration="1.614525933s" totalImagesPullingTime="178.964519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.613316384 +0000 UTC m=+629.742550307" watchObservedRunningTime="2026-10-30 19:45:05.614525933 +0000 UTC m=+629.743759851" Oct 30 19:45:06.081412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:06.081382 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:06.400225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:06.400197 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:06.528085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:06.528059 4959 scope.go:122] "RemoveContainer" containerID="e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c" Oct 30 19:45:06.528469 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:06.528246 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-hpbwp_odh-ai-gateway-infra(014b7614-80f6-46bb-b643-b142e1b1b773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" Oct 30 19:45:07.826533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:07.826487 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.041189 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:10.041154 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:10.053734 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:10.053710 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:10.554895 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:10.554873 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5npp4" Oct 30 19:45:14.400342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:14.400311 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:14.412843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:14.412822 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:14.571098 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:14.571073 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" Oct 30 19:45:16.064595 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:16.064560 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:45:18.990058 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:18.990021 4959 scope.go:122] "RemoveContainer" containerID="e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c" Oct 30 19:45:19.587313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:19.587225 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerStarted","Data":"cd7e9044a664922bf6b222b7f78f08cdd4d176756435f73c6c1a1d73dbd13c1b"} Oct 30 19:45:20.593766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:20.593736 4959 generic.go:356] "Generic (PLEG): container finished" podID="014b7614-80f6-46bb-b643-b142e1b1b773" containerID="cd7e9044a664922bf6b222b7f78f08cdd4d176756435f73c6c1a1d73dbd13c1b" exitCode=60 Oct 30 19:45:20.594219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:20.593811 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerDied","Data":"cd7e9044a664922bf6b222b7f78f08cdd4d176756435f73c6c1a1d73dbd13c1b"} Oct 30 19:45:20.594219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:20.593868 4959 scope.go:122] "RemoveContainer" containerID="e282971745ce059c583f7abdb020309d74714ff95566d3007da00b1669ecbb9c" Oct 30 19:45:20.594219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:20.594165 4959 scope.go:122] "RemoveContainer" containerID="cd7e9044a664922bf6b222b7f78f08cdd4d176756435f73c6c1a1d73dbd13c1b" Oct 30 19:45:20.594410 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:20.594384 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-hpbwp_odh-ai-gateway-infra(014b7614-80f6-46bb-b643-b142e1b1b773)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" Oct 30 19:45:20.648234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:20.648201 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:21.727012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.726986 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:21.824133 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.824098 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\" (UniqueName: \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\") pod \"014b7614-80f6-46bb-b643-b142e1b1b773\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " Oct 30 19:45:21.824297 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.824163 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\") pod \"014b7614-80f6-46bb-b643-b142e1b1b773\" (UID: \"014b7614-80f6-46bb-b643-b142e1b1b773\") " Oct 30 19:45:21.824540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.824507 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca" pod "014b7614-80f6-46bb-b643-b142e1b1b773" (UID: "014b7614-80f6-46bb-b643-b142e1b1b773"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.826404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.826381 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr" pod "014b7614-80f6-46bb-b643-b142e1b1b773" (UID: "014b7614-80f6-46bb-b643-b142e1b1b773"). InnerVolumeSpecName "kube-api-access-fkgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.925027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.925000 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgkr\" (UniqueName: \"kubernetes.io/projected/014b7614-80f6-46bb-b643-b142e1b1b773-kube-api-access-fkgkr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:45:21.925027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:21.925029 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/014b7614-80f6-46bb-b643-b142e1b1b773-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.503228 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.503941 4959 cpu_manager.go:414] "containerMap: removing container" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.503961 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.503979 4959 cpu_manager.go:414] "containerMap: removing container" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.503986 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.504172 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.505783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.504197 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.508020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.507990 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.533599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.533572 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-94wtv\"" Oct 30 19:45:22.548942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.548914 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 19:45:22.557102 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.557076 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:45:22.557638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.557620 4959 cpu_manager.go:414] "containerMap: removing container" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.557700 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.557642 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.557763 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.557750 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" containerName="cleanup" Oct 30 19:45:22.561173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.561149 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.593550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.593517 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:45:22.605301 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.605276 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" Oct 30 19:45:22.605444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.605304 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp" event={"ID":"014b7614-80f6-46bb-b643-b142e1b1b773","Type":"ContainerDied","Data":"aa534c04c63172ca951c95aa8ef9b09491fb0596588e56cf96b9258af357e6bb"} Oct 30 19:45:22.605444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.605363 4959 scope.go:122] "RemoveContainer" containerID="cd7e9044a664922bf6b222b7f78f08cdd4d176756435f73c6c1a1d73dbd13c1b" Oct 30 19:45:22.632686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632664 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632694 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632716 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632732 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632837 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632866 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.632935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632884 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.633082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.632930 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnp8\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.633082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.633039 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.705930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.705890 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:22.710659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.710635 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-hpbwp"] Oct 30 19:45:22.733867 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733826 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.733867 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733869 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733891 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733911 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733927 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733944 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733972 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.733993 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734021 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734203 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734247 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734313 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnp8\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734348 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.734407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734390 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734425 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734519 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734575 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734630 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734698 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734775 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bw9\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.734872 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.735282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.735058 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.736486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.736464 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.736600 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.736577 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.762374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.762301 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.762374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.762296 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnp8\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lr88d\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.820358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.820323 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:22.835930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.835903 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.835937 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.835955 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.835974 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836018 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836060 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836081 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836114 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836148 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bw9\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836349 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836372 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836422 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.836804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.836782 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.837617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.837593 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.838803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.838781 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.838912 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.838896 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.866335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.866300 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.866532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.866400 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bw9\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-stmx4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:22.874148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:22.874120 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:23.137554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:23.137525 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 19:45:23.138512 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:23.138459 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a6dc58_5919_49b9_9165_466126251f9a.slice/crio-4bdee8b74be08c03a1b55987e26aab3a2a27a5c1ebe3fef9412004bc37134177 WatchSource:0}: Error finding container 4bdee8b74be08c03a1b55987e26aab3a2a27a5c1ebe3fef9412004bc37134177: Status 404 returned error can't find the container with id 4bdee8b74be08c03a1b55987e26aab3a2a27a5c1ebe3fef9412004bc37134177 Oct 30 19:45:23.181143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:23.181121 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:45:23.182660 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:23.182633 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c4fcbd_d5f6_405d_9f04_fa1c2f3964e4.slice/crio-d3619d2066009fe57718301e23855d03a052aa023403956f42f5b6c31cc91167 WatchSource:0}: Error finding container d3619d2066009fe57718301e23855d03a052aa023403956f42f5b6c31cc91167: Status 404 returned error can't find the container with id d3619d2066009fe57718301e23855d03a052aa023403956f42f5b6c31cc91167 Oct 30 19:45:23.612615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:23.612520 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" event={"ID":"b9a6dc58-5919-49b9-9165-466126251f9a","Type":"ContainerStarted","Data":"4bdee8b74be08c03a1b55987e26aab3a2a27a5c1ebe3fef9412004bc37134177"} Oct 30 19:45:23.615137 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:23.615094 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" event={"ID":"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4","Type":"ContainerStarted","Data":"d3619d2066009fe57718301e23855d03a052aa023403956f42f5b6c31cc91167"} Oct 30 19:45:23.996376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:23.996342 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014b7614-80f6-46bb-b643-b142e1b1b773" path="/var/lib/kubelet/pods/014b7614-80f6-46bb-b643-b142e1b1b773/volumes" Oct 30 19:45:25.773226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.773185 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:25.773611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.773263 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:25.773611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.773298 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:25.966446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.966404 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:25.966584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.966519 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:25.966584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:25.966563 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:45:26.629824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.629787 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" event={"ID":"b9a6dc58-5919-49b9-9165-466126251f9a","Type":"ContainerStarted","Data":"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9"} Oct 30 19:45:26.631305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.631278 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" event={"ID":"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4","Type":"ContainerStarted","Data":"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d"} Oct 30 19:45:26.714311 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.714244 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podStartSLOduration=1.888983143 podStartE2EDuration="4.714231913s" totalImagesPullingTime="2.82524877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.70076652 +0000 UTC m=+650.830000465" watchObservedRunningTime="2026-10-30 19:45:26.714231913 +0000 UTC m=+650.843465842" Oct 30 19:45:26.752990 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.752928 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podStartSLOduration=2.165130954 podStartE2EDuration="4.752907347s" totalImagesPullingTime="2.587776393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:26.752758891 +0000 UTC m=+650.881992813" watchObservedRunningTime="2026-10-30 19:45:26.752907347 +0000 UTC m=+650.882141269" Oct 30 19:45:26.820504 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.820467 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:26.878100 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.878053 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:26.886030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.885997 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:26.886307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:26.886264 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:27.589519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.587946 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.605382 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.605352 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9874ff978-sr9zq"] Oct 30 19:45:27.608880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.608865 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.628165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.628139 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:45:27.641541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.641518 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhvjj\"" Oct 30 19:45:27.641656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.641546 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:45:27.641656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.641581 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:45:27.642991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.642969 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9874ff978-sr9zq"] Oct 30 19:45:27.705563 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.705527 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr7c\" (UniqueName: \"kubernetes.io/projected/a0eaa649-8275-4d68-8e6a-3dd606034036-kube-api-access-dwr7c\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.705749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.705608 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0eaa649-8275-4d68-8e6a-3dd606034036-tls-cert\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.806968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.806931 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr7c\" (UniqueName: \"kubernetes.io/projected/a0eaa649-8275-4d68-8e6a-3dd606034036-kube-api-access-dwr7c\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.807152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.806980 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0eaa649-8275-4d68-8e6a-3dd606034036-tls-cert\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.809937 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.809915 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a0eaa649-8275-4d68-8e6a-3dd606034036-tls-cert\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.822883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.822859 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:27.823256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.823235 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:27.828872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.828843 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr7c\" (UniqueName: \"kubernetes.io/projected/a0eaa649-8275-4d68-8e6a-3dd606034036-kube-api-access-dwr7c\") pod \"authorino-9874ff978-sr9zq\" (UID: \"a0eaa649-8275-4d68-8e6a-3dd606034036\") " pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:27.879575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.878921 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:27.879575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.878983 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:27.938402 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:27.938373 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9874ff978-sr9zq" Oct 30 19:45:28.122298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.122267 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:28.127504 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.127417 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.144045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.144021 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6zrfq\"" Oct 30 19:45:28.144300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.144286 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:45:28.144840 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.144828 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:45:28.144907 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.144897 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:45:28.151924 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.151901 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:45:28.200103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.200034 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:28.209790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.209588 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgd8w\" (UniqueName: \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.209790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.209622 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.239128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.239093 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.302234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.302191 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9874ff978-sr9zq"] Oct 30 19:45:28.303087 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:28.303056 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0eaa649_8275_4d68_8e6a_3dd606034036.slice/crio-0f1c78c1c322f81d8ed92a16938e3ab36a60c0d51a870ba25283d12025c363a4 WatchSource:0}: Error finding container 0f1c78c1c322f81d8ed92a16938e3ab36a60c0d51a870ba25283d12025c363a4: Status 404 returned error can't find the container with id 0f1c78c1c322f81d8ed92a16938e3ab36a60c0d51a870ba25283d12025c363a4 Oct 30 19:45:28.310179 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.310155 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgd8w\" (UniqueName: \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.310262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.310191 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.313149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.313124 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.331890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.331863 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgd8w\" (UniqueName: \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\") pod \"ai-gateway-controller-66dd76bf94-6tj8k\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.387593 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.386173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.443282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.443221 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:28.641370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.641286 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9874ff978-sr9zq" event={"ID":"a0eaa649-8275-4d68-8e6a-3dd606034036","Type":"ContainerStarted","Data":"0f1c78c1c322f81d8ed92a16938e3ab36a60c0d51a870ba25283d12025c363a4"} Oct 30 19:45:28.797565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.796100 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:28.801107 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:28.801006 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dff0610_9610_4591_878a_93b46e0c8330.slice/crio-ce8a2410d1a40dcbb6fc6a5914c877640da42ddc66d61b6c489e9079f3d1a1f2 WatchSource:0}: Error finding container ce8a2410d1a40dcbb6fc6a5914c877640da42ddc66d61b6c489e9079f3d1a1f2: Status 404 returned error can't find the container with id ce8a2410d1a40dcbb6fc6a5914c877640da42ddc66d61b6c489e9079f3d1a1f2 Oct 30 19:45:28.820840 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.820803 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:28.820968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.820866 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:28.891044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.890943 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:28.891044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:28.891007 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:29.703821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.703607 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9874ff978-sr9zq" event={"ID":"a0eaa649-8275-4d68-8e6a-3dd606034036","Type":"ContainerStarted","Data":"f0665c1c93738d8ba9aa8d04fad6f3be59e73bfaa8dfdf3cb47486917c162315"} Oct 30 19:45:29.707776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.707731 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" event={"ID":"2dff0610-9610-4591-878a-93b46e0c8330","Type":"ContainerStarted","Data":"ce8a2410d1a40dcbb6fc6a5914c877640da42ddc66d61b6c489e9079f3d1a1f2"} Oct 30 19:45:29.770521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.762425 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-9874ff978-sr9zq" podStartSLOduration=1.940076406 podStartE2EDuration="2.762407134s" totalImagesPullingTime="822.330728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:29.762123412 +0000 UTC m=+653.891357334" watchObservedRunningTime="2026-10-30 19:45:29.762407134 +0000 UTC m=+653.891641055" Oct 30 19:45:29.820943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.820862 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:29.820943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.820909 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:29.894520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.877949 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:29.894520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:29.878013 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:30.820916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:30.820876 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:30.821121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:30.820937 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:30.878518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:30.878041 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:30.878518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:30.878106 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:31.821249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:31.821216 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:31.821698 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:31.821262 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:31.892519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:31.889607 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:31.892519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:31.889671 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:32.729803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.729764 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" event={"ID":"2dff0610-9610-4591-878a-93b46e0c8330","Type":"ContainerStarted","Data":"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3"} Oct 30 19:45:32.730134 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.729875 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:32.782109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.782029 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" podStartSLOduration=1.938785652 podStartE2EDuration="4.782009353s" totalImagesPullingTime="2.843223701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:32.781571063 +0000 UTC m=+656.910804985" watchObservedRunningTime="2026-10-30 19:45:32.782009353 +0000 UTC m=+656.911243277" Oct 30 19:45:32.821883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.821846 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 30 19:45:32.822405 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.821912 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 30 19:45:32.875458 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.875409 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:45:32.875709 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:32.875480 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 30 19:45:34.357312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.357165 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:34.475250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.475202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:34.519285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.519252 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2"] Oct 30 19:45:34.523147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.523115 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.557543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.557465 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2"] Oct 30 19:45:34.570651 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.570621 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:45:34.570957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.570935 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q2ntt\"" Oct 30 19:45:34.587971 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.587930 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgv4\" (UniqueName: \"kubernetes.io/projected/561f2758-1a08-4cbb-b4cd-22080c44c16f-kube-api-access-5rgv4\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.588180 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.588016 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f2758-1a08-4cbb-b4cd-22080c44c16f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.688850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.688801 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgv4\" (UniqueName: \"kubernetes.io/projected/561f2758-1a08-4cbb-b4cd-22080c44c16f-kube-api-access-5rgv4\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.689050 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.688887 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f2758-1a08-4cbb-b4cd-22080c44c16f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.689836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.689802 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/561f2758-1a08-4cbb-b4cd-22080c44c16f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.711575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.711543 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgv4\" (UniqueName: \"kubernetes.io/projected/561f2758-1a08-4cbb-b4cd-22080c44c16f-kube-api-access-5rgv4\") pod \"payload-pre-processing-7b98bb678c-7zvc2\" (UID: \"561f2758-1a08-4cbb-b4cd-22080c44c16f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.821324 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.821269 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:34.821551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.821346 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:34.842665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.842632 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:34.877448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.876914 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:34.877448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:34.876974 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:35.162362 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.162254 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2"] Oct 30 19:45:35.182818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.180840 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:45:35.749094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.749054 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" event={"ID":"561f2758-1a08-4cbb-b4cd-22080c44c16f","Type":"ContainerStarted","Data":"6b805edb1d6c4252d5a1e0665fd421b674d5b8d5b486e037e8b0658841106904"} Oct 30 19:45:35.863786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.862859 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:35.863786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.862926 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:35.920438 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.919665 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:35.920438 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:35.919730 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.914549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:36.913617 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lr88d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.914549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:36.913687 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.967520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:36.966627 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-stmx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.967520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:36.966695 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.315750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.315146 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:37.315750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.315631 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:37.316556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.316536 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:45:37.394522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.393868 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:37.400521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.394767 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:37.400521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:37.395828 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 19:45:38.810981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:38.810871 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" event={"ID":"561f2758-1a08-4cbb-b4cd-22080c44c16f","Type":"ContainerStarted","Data":"dbe47d37956e76d37f9fcd4e64f4f1214bbfebabe62340399f0b7babb4b71638"} Oct 30 19:45:38.811453 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:38.811321 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:38.859624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:38.859576 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" podStartSLOduration=1.610080415 podStartE2EDuration="4.859563481s" totalImagesPullingTime="3.249483066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:38.857527972 +0000 UTC m=+662.986761894" watchObservedRunningTime="2026-10-30 19:45:38.859563481 +0000 UTC m=+662.988797402" Oct 30 19:45:43.739414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:43.739374 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:44.052877 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.052798 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd"] Oct 30 19:45:44.057721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.057686 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.085226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.085193 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd"] Oct 30 19:45:44.092954 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.092924 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f103348-8709-416f-af44-07d3eee0737d-metrics-certs\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.093140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.093053 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdfk\" (UniqueName: \"kubernetes.io/projected/5f103348-8709-416f-af44-07d3eee0737d-kube-api-access-wxdfk\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.194589 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.194548 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f103348-8709-416f-af44-07d3eee0737d-metrics-certs\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.194766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.194603 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdfk\" (UniqueName: \"kubernetes.io/projected/5f103348-8709-416f-af44-07d3eee0737d-kube-api-access-wxdfk\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.197193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.197167 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f103348-8709-416f-af44-07d3eee0737d-metrics-certs\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.216803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.216775 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdfk\" (UniqueName: \"kubernetes.io/projected/5f103348-8709-416f-af44-07d3eee0737d-kube-api-access-wxdfk\") pod \"ai-gateway-controller-57ff5bcdf7-twvfd\" (UID: \"5f103348-8709-416f-af44-07d3eee0737d\") " pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.372720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.372641 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:44.677421 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.677394 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd"] Oct 30 19:45:44.678385 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:45:44.678354 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f103348_8709_416f_af44_07d3eee0737d.slice/crio-4a63ee53b896bce51ff5e33363979c2f3ccc512a8fc8cd55c728731c5ed18d7f WatchSource:0}: Error finding container 4a63ee53b896bce51ff5e33363979c2f3ccc512a8fc8cd55c728731c5ed18d7f: Status 404 returned error can't find the container with id 4a63ee53b896bce51ff5e33363979c2f3ccc512a8fc8cd55c728731c5ed18d7f Oct 30 19:45:44.839107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:44.839073 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" event={"ID":"5f103348-8709-416f-af44-07d3eee0737d","Type":"ContainerStarted","Data":"4a63ee53b896bce51ff5e33363979c2f3ccc512a8fc8cd55c728731c5ed18d7f"} Oct 30 19:45:45.845342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:45.845306 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" event={"ID":"5f103348-8709-416f-af44-07d3eee0737d","Type":"ContainerStarted","Data":"daf2bf89f4ed5bafdabf799e786b5686eb1c1092f3d76ccf53e77ea8fbeb1824"} Oct 30 19:45:45.845805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:45.845438 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:45.890506 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:45.890432 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" podStartSLOduration=1.571609417 podStartE2EDuration="1.890410919s" totalImagesPullingTime="318.801502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:45.888996849 +0000 UTC m=+670.018230769" watchObservedRunningTime="2026-10-30 19:45:45.890410919 +0000 UTC m=+670.019644864" Oct 30 19:45:46.065486 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:46.065452 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:45:50.822765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:50.822737 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7zvc2" Oct 30 19:45:56.853055 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:56.853017 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-57ff5bcdf7-twvfd" Oct 30 19:45:56.941381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:56.941348 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:56.941909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:56.941634 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" podUID="2dff0610-9610-4591-878a-93b46e0c8330" containerName="manager" containerID="cri-o://ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3" gracePeriod=30 Oct 30 19:45:57.187131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.187099 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:57.318034 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.318001 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\" (UniqueName: \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\") pod \"2dff0610-9610-4591-878a-93b46e0c8330\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " Oct 30 19:45:57.318204 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.318058 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\") pod \"2dff0610-9610-4591-878a-93b46e0c8330\" (UID: \"2dff0610-9610-4591-878a-93b46e0c8330\") " Oct 30 19:45:57.320463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.320431 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs" pod "2dff0610-9610-4591-878a-93b46e0c8330" (UID: "2dff0610-9610-4591-878a-93b46e0c8330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:57.320463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.320434 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w" pod "2dff0610-9610-4591-878a-93b46e0c8330" (UID: "2dff0610-9610-4591-878a-93b46e0c8330"). InnerVolumeSpecName "kube-api-access-dgd8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:57.419209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.419179 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgd8w\" (UniqueName: \"kubernetes.io/projected/2dff0610-9610-4591-878a-93b46e0c8330-kube-api-access-dgd8w\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:45:57.419209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.419206 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dff0610-9610-4591-878a-93b46e0c8330-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:45:57.900186 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.900157 4959 generic.go:356] "Generic (PLEG): container finished" podID="2dff0610-9610-4591-878a-93b46e0c8330" containerID="ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3" exitCode=0 Oct 30 19:45:57.900614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.900206 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" Oct 30 19:45:57.900614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.900238 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" event={"ID":"2dff0610-9610-4591-878a-93b46e0c8330","Type":"ContainerDied","Data":"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3"} Oct 30 19:45:57.900614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.900290 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6tj8k" event={"ID":"2dff0610-9610-4591-878a-93b46e0c8330","Type":"ContainerDied","Data":"ce8a2410d1a40dcbb6fc6a5914c877640da42ddc66d61b6c489e9079f3d1a1f2"} Oct 30 19:45:57.900614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.900312 4959 scope.go:122] "RemoveContainer" containerID="ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3" Oct 30 19:45:57.909092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.909075 4959 scope.go:122] "RemoveContainer" containerID="ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3" Oct 30 19:45:57.909371 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:45:57.909353 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3\": container with ID starting with ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3 not found: ID does not exist" containerID="ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3" Oct 30 19:45:57.909421 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.909381 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3"} err="failed to get container status \"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3\": rpc error: code = NotFound desc = could not find container \"ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3\": container with ID starting with ec5983154ba691120e976b793d715e03176b7bcb762490760085d90e295f29f3 not found: ID does not exist" Oct 30 19:45:57.961332 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.961300 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:57.974254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.974225 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6tj8k"] Oct 30 19:45:57.994651 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:57.994621 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dff0610-9610-4591-878a-93b46e0c8330" path="/var/lib/kubelet/pods/2dff0610-9610-4591-878a-93b46e0c8330/volumes" Oct 30 19:45:58.719383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:58.719351 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:59.277670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:45:59.277635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.946735 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:00.946697 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.063579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:01.063546 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.574506 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:02.574469 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.744470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:02.744441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:16.065188 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:46:16.065153 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:46:25.600966 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:25.600930 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:25.727652 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:25.727621 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.083783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.083750 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 19:46:26.084126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.084114 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2dff0610-9610-4591-878a-93b46e0c8330" containerName="manager" Oct 30 19:46:26.084172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.084128 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dff0610-9610-4591-878a-93b46e0c8330" containerName="manager" Oct 30 19:46:26.084207 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.084185 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dff0610-9610-4591-878a-93b46e0c8330" containerName="manager" Oct 30 19:46:26.087337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.087320 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.106866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.106845 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-284bf6-openshift-default-dockercfg-fq2xw\"" Oct 30 19:46:26.120410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.120383 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 19:46:26.143677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.143656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.174862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.174837 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.174976 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.174874 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.174976 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.174894 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.174976 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.174911 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.175080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.175005 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xhd\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.175080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.175032 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.175148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.175103 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.175148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.175121 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.175219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.175143 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.230033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.229997 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 19:46:26.234025 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.234000 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.250202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.250172 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f349a1-openshift-default-dockercfg-n7zxx\"" Oct 30 19:46:26.257889 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.257862 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 19:46:26.260988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.260958 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.277964 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.277936 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.278176 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278420 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278397 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4qs\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.278601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278582 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278688 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278734 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.278901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278860 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278896 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278896 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.278987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278925 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.278987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.278952 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.279197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279175 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xhd\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279292 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279223 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279511 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279600 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279555 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279664 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279639 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279665 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.279721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279691 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.279838 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279719 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.279838 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279827 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.279938 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279859 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.279938 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.279884 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.280327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.280267 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.282193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.282168 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.282689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.282668 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.302683 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.302662 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xhd\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.302773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.302697 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.373281 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.373204 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 19:46:26.377909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.377888 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.380614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380591 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380619 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380638 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380656 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380705 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380782 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4qs\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380814 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380852 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.380958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.381167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.380988 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.381437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.381419 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.381689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.381668 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.382277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.382050 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.382277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.382150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.382415 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.382358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.384448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.384428 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.384547 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.384472 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.400929 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.400909 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:26.403633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.403467 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 19:46:26.407074 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.407056 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4qs\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.407212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.407191 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.438264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.438216 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-01f7d3-openshift-default-dockercfg-mt2rl\"" Oct 30 19:46:26.481836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.481810 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5pw\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.481978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.481856 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.481978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.481938 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.481978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.481975 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.482270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.482009 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.482270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.482048 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.482270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.482082 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.482270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.482131 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.482270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.482156 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.508662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.508632 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.548064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.547803 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:26.583003 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.582973 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5pw\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583038 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583073 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583101 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583129 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583153 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583221 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583255 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583591 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583486 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583682 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583661 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583677 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.583821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.583802 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.584854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.584822 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.586173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.585954 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.586305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.586287 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.605281 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.605253 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5pw\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.608860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.608834 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.633767 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.633742 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.705707 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.705679 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:26.722788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.722766 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 19:46:26.724435 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:26.724410 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d7759f_78c0_4427_8451_fe3022de4942.slice/crio-9ba410b13c963b502a3b7980f09bd414527642e7a7ceb28f1cbed45a9db3123f WatchSource:0}: Error finding container 9ba410b13c963b502a3b7980f09bd414527642e7a7ceb28f1cbed45a9db3123f: Status 404 returned error can't find the container with id 9ba410b13c963b502a3b7980f09bd414527642e7a7ceb28f1cbed45a9db3123f Oct 30 19:46:26.726720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.726688 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.726810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.726752 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.726810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.726779 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.744069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.744041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.859769 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.859737 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 19:46:26.863437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.863411 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887146 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887191 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887280 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887413 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887454 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgjm\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887589 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887484 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887667 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887706 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.887752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.887738 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.893444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.893424 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-109cfe-openshift-default-dockercfg-4jtdp\"" Oct 30 19:46:26.899747 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:26.899720 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850ff4e7_92c1_435f_86a6_54416b38c49d.slice/crio-8bc3ce4c3357860dbda3605d42aa6adc9d6b36e529bd307088757169246ee96d WatchSource:0}: Error finding container 8bc3ce4c3357860dbda3605d42aa6adc9d6b36e529bd307088757169246ee96d: Status 404 returned error can't find the container with id 8bc3ce4c3357860dbda3605d42aa6adc9d6b36e529bd307088757169246ee96d Oct 30 19:46:26.900006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.899988 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 19:46:26.902234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.902206 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.902317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.902269 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.902317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.902300 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:26.923978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.923812 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.926547 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.924554 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 19:46:26.989229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989194 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989238 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgjm\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989451 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989430 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989557 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989594 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989626 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989657 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989685 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989711 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.989937 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.989842 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.990052 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.990032 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.990125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.990103 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.990265 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.990230 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.990374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.990294 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.992524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.992473 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:26.992631 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:26.992614 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:27.027056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.027010 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerStarted","Data":"a456e0a59c5d80f86d93626f4839293350ec3e6ec38a1317d1c4558349801c1f"} Oct 30 19:46:27.027236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.027078 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerStarted","Data":"8bc3ce4c3357860dbda3605d42aa6adc9d6b36e529bd307088757169246ee96d"} Oct 30 19:46:27.028878 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.028852 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerStarted","Data":"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab"} Oct 30 19:46:27.029024 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.028903 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerStarted","Data":"9ba410b13c963b502a3b7980f09bd414527642e7a7ceb28f1cbed45a9db3123f"} Oct 30 19:46:27.032146 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.032116 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:27.035285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.035261 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgjm\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:27.055668 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:27.055626 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a98ac8a_04b4_4aac_8b5d_a366e28c0ba6.slice/crio-6a4a0c4d6cc4064e2310eaca1e96eff085ec500fec3557efe70df6fbbf9572eb WatchSource:0}: Error finding container 6a4a0c4d6cc4064e2310eaca1e96eff085ec500fec3557efe70df6fbbf9572eb: Status 404 returned error can't find the container with id 6a4a0c4d6cc4064e2310eaca1e96eff085ec500fec3557efe70df6fbbf9572eb Oct 30 19:46:27.059525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.056347 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 19:46:27.068538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.068035 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.068538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.068118 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.068538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.068156 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.096246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.096202 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podStartSLOduration=1.09618949 podStartE2EDuration="1.09618949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:27.093680675 +0000 UTC m=+711.222914598" watchObservedRunningTime="2026-10-30 19:46:27.09618949 +0000 UTC m=+711.225423408" Oct 30 19:46:27.111516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.111457 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.175989 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.175915 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podStartSLOduration=1.175893594 podStartE2EDuration="1.175893594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:27.17390802 +0000 UTC m=+711.303141942" watchObservedRunningTime="2026-10-30 19:46:27.175893594 +0000 UTC m=+711.305127516" Oct 30 19:46:27.179073 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.178847 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:27.232134 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.232070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.358826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.358791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.402918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.402835 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:27.409975 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.409945 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:27.495204 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.495182 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 19:46:27.496884 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:27.496858 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd379af6f_5f5c_4a9f_abcb_e14e5fdb044a.slice/crio-26133b90ed2cebc7be791e9b7a1e81f00d0f19a11e007370129bcc8b3a5ece3d WatchSource:0}: Error finding container 26133b90ed2cebc7be791e9b7a1e81f00d0f19a11e007370129bcc8b3a5ece3d: Status 404 returned error can't find the container with id 26133b90ed2cebc7be791e9b7a1e81f00d0f19a11e007370129bcc8b3a5ece3d Oct 30 19:46:27.499477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.499449 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.499632 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.499553 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.499632 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.499607 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:27.548544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.548516 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:27.553747 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.553723 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:27.665555 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:27.665449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:28.035275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.035191 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerStarted","Data":"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9"} Oct 30 19:46:28.035275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.035248 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerStarted","Data":"26133b90ed2cebc7be791e9b7a1e81f00d0f19a11e007370129bcc8b3a5ece3d"} Oct 30 19:46:28.036921 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.036890 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerStarted","Data":"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34"} Oct 30 19:46:28.037053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.036956 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerStarted","Data":"6a4a0c4d6cc4064e2310eaca1e96eff085ec500fec3557efe70df6fbbf9572eb"} Oct 30 19:46:28.037565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.037425 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:28.037565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.037523 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:28.038233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.038211 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:28.038325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.038286 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:28.095343 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.095289 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podStartSLOduration=2.09527535 podStartE2EDuration="2.09527535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:28.093815274 +0000 UTC m=+712.223049196" watchObservedRunningTime="2026-10-30 19:46:28.09527535 +0000 UTC m=+712.224509340" Oct 30 19:46:28.179866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.179833 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:28.184947 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.184924 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:28.195474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.195417 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podStartSLOduration=2.19540023 podStartE2EDuration="2.19540023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:28.193569358 +0000 UTC m=+712.322803281" watchObservedRunningTime="2026-10-30 19:46:28.19540023 +0000 UTC m=+712.324634152" Oct 30 19:46:28.706094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.706060 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:28.712112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:28.712082 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:29.042075 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:29.041988 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:29.042238 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:29.042197 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:29.043160 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:29.043126 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:46:29.043268 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:29.043253 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:46:29.912766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:29.912732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.050309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.050278 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.184951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.184878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.301411 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.301380 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.474787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.474704 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.796850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.796769 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.922096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:30.922070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.078039 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:31.077967 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.259486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:31.259454 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.970395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:31.970359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.108305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.108276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.202466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.202437 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 19:46:32.207112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.207095 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.227887 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.227807 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-f349a1-dockercfg-2brnv\"" Oct 30 19:46:32.241566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.241546 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-f349a1\"" Oct 30 19:46:32.243190 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.243171 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 19:46:32.244931 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.244913 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fq6\" (UniqueName: \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.245006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.244945 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.323563 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.319147 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 19:46:32.329680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.329653 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 19:46:32.330033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.329961 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.346169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.346141 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fq6\" (UniqueName: \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.346350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.346187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.347111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.347085 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.374339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.374310 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fq6\" (UniqueName: \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\") pod \"payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.447663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.447631 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.447828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.447690 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5px\" (UniqueName: \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.519327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.519257 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:32.549349 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.549316 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.549481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.549406 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5px\" (UniqueName: \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.550038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.550019 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.577464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.577441 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5px\" (UniqueName: \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\") pod \"payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.651525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.651437 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:32.820779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.820754 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 19:46:32.822115 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:32.822088 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151b789c_59c3_4f70_8b36_d7203dbed073.slice/crio-7868caaa6b96234e92cf4a6b045faee334c397a73f78360fbb095d0d7a2a9944 WatchSource:0}: Error finding container 7868caaa6b96234e92cf4a6b045faee334c397a73f78360fbb095d0d7a2a9944: Status 404 returned error can't find the container with id 7868caaa6b96234e92cf4a6b045faee334c397a73f78360fbb095d0d7a2a9944 Oct 30 19:46:32.956928 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:32.956897 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c14d5fa_3809_4428_8be6_0a5501383465.slice/crio-cea73a4249f04061b184890b7ca3949d79935aaee02e187bcf669e23e32bd007 WatchSource:0}: Error finding container cea73a4249f04061b184890b7ca3949d79935aaee02e187bcf669e23e32bd007: Status 404 returned error can't find the container with id cea73a4249f04061b184890b7ca3949d79935aaee02e187bcf669e23e32bd007 Oct 30 19:46:32.957557 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:32.957436 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 19:46:33.068714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.068679 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" event={"ID":"151b789c-59c3-4f70-8b36-d7203dbed073","Type":"ContainerStarted","Data":"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690"} Oct 30 19:46:33.069360 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.068846 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:33.069360 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.068878 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" event={"ID":"151b789c-59c3-4f70-8b36-d7203dbed073","Type":"ContainerStarted","Data":"7868caaa6b96234e92cf4a6b045faee334c397a73f78360fbb095d0d7a2a9944"} Oct 30 19:46:33.071210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.071180 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" event={"ID":"1c14d5fa-3809-4428-8be6-0a5501383465","Type":"ContainerStarted","Data":"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad"} Oct 30 19:46:33.071367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.071227 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" event={"ID":"1c14d5fa-3809-4428-8be6-0a5501383465","Type":"ContainerStarted","Data":"cea73a4249f04061b184890b7ca3949d79935aaee02e187bcf669e23e32bd007"} Oct 30 19:46:33.071367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.071325 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:33.075103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.075081 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.117508 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.117429 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" podStartSLOduration=1.117409135 podStartE2EDuration="1.117409135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:33.11674794 +0000 UTC m=+717.245981861" watchObservedRunningTime="2026-10-30 19:46:33.117409135 +0000 UTC m=+717.246643094" Oct 30 19:46:33.190435 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.190402 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.245607 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.245558 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" podStartSLOduration=1.245541501 podStartE2EDuration="1.245541501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:33.16087538 +0000 UTC m=+717.290109300" watchObservedRunningTime="2026-10-30 19:46:33.245541501 +0000 UTC m=+717.374775420" Oct 30 19:46:33.246867 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.246849 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 19:46:33.250382 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.250367 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.271215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.271186 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 19:46:33.275787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.275761 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-284bf6\"" Oct 30 19:46:33.312066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.308964 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-284bf6-dockercfg-67ggm\"" Oct 30 19:46:33.315264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.315236 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.361343 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.361244 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.361548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.361479 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wnl\" (UniqueName: \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.454745 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.454710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.462895 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.462872 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.462991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.462977 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wnl\" (UniqueName: \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.463566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.463551 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.490322 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.490295 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wnl\" (UniqueName: \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\") pod \"payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.564020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.563995 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:33.611529 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.611429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.760841 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.760812 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.892530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:33.890932 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 19:46:34.067197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.067164 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.078573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.078536 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" event={"ID":"d3a8512e-1144-4d81-a115-51de804273cc","Type":"ContainerStarted","Data":"ece60ccd8c6bb99fb191b6af8f73697bc478351f1554e1348cc6b2150d08f7a1"} Oct 30 19:46:34.079056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.078921 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:34.079056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.078959 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" event={"ID":"d3a8512e-1144-4d81-a115-51de804273cc","Type":"ContainerStarted","Data":"15f9822c5379601f1324f74beb0b9f5706da0591e60abb082f90f580f33b3821"} Oct 30 19:46:34.107425 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.107402 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 19:46:34.110943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.110930 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.131701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.131681 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:46:34.133578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.133554 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:46:34.133685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.133601 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-284bf6-serving-cert\"" Oct 30 19:46:34.133685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.133646 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcrrd\"" Oct 30 19:46:34.137353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.137271 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 19:46:34.145333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.145270 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:46:34.146692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.146644 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" podStartSLOduration=1.14663239 podStartE2EDuration="1.14663239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:34.145307672 +0000 UTC m=+718.274541593" watchObservedRunningTime="2026-10-30 19:46:34.14663239 +0000 UTC m=+718.275866311" Oct 30 19:46:34.168567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.168541 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhpl\" (UniqueName: \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.168703 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.168613 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.168778 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.168751 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.169150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.168959 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.201597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.201568 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.269779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.269754 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.269894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.269792 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhpl\" (UniqueName: \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.269894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.269825 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.269894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.269846 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.270378 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.270359 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.272277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.272259 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.272323 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.272311 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.296333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.296304 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhpl\" (UniqueName: \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\") pod \"maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.326344 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.326323 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.330377 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.330343 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 19:46:34.334127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.334109 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.366775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.365082 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6c4dd1-dockercfg-hvfnv\"" Oct 30 19:46:34.378489 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.377723 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6c4dd1\"" Oct 30 19:46:34.378489 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.378256 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fj8\" (UniqueName: \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.378489 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.378331 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.424794 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.423473 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:34.474563 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.474527 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 19:46:34.479689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.479658 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.479839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.479823 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fj8\" (UniqueName: \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.480662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.480636 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.511300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.511271 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fj8\" (UniqueName: \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\") pod \"payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.647293 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.647259 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:34.737213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.737186 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 19:46:34.738299 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:34.738275 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d94575_6bf2_45a5_a7f2_f72dd9a14ae8.slice/crio-7d51b44b5e0d44fb9550e2df39a024a1a37948d5ff0c37aa70d47e75a09f0295 WatchSource:0}: Error finding container 7d51b44b5e0d44fb9550e2df39a024a1a37948d5ff0c37aa70d47e75a09f0295: Status 404 returned error can't find the container with id 7d51b44b5e0d44fb9550e2df39a024a1a37948d5ff0c37aa70d47e75a09f0295 Oct 30 19:46:34.953464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:34.953437 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 19:46:34.953655 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:34.953622 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ad2e0e_8576_4f37_931f_ce70b65e4e35.slice/crio-e3a84cbde477e791be8270e1701dce11e4bae92baff2f2fbbc9d5dd2e1116980 WatchSource:0}: Error finding container e3a84cbde477e791be8270e1701dce11e4bae92baff2f2fbbc9d5dd2e1116980: Status 404 returned error can't find the container with id e3a84cbde477e791be8270e1701dce11e4bae92baff2f2fbbc9d5dd2e1116980 Oct 30 19:46:35.084143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.084118 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" event={"ID":"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8","Type":"ContainerStarted","Data":"7d51b44b5e0d44fb9550e2df39a024a1a37948d5ff0c37aa70d47e75a09f0295"} Oct 30 19:46:35.085554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.085527 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" event={"ID":"21ad2e0e-8576-4f37-931f-ce70b65e4e35","Type":"ContainerStarted","Data":"09375439bb2371942b295f1181747ab437939719afe2fc60f9f8512e67d30fbe"} Oct 30 19:46:35.085644 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.085577 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" event={"ID":"21ad2e0e-8576-4f37-931f-ce70b65e4e35","Type":"ContainerStarted","Data":"e3a84cbde477e791be8270e1701dce11e4bae92baff2f2fbbc9d5dd2e1116980"} Oct 30 19:46:35.085791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.085775 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:35.137899 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.137855 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" podStartSLOduration=1.13781884 podStartE2EDuration="1.13781884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.136891169 +0000 UTC m=+719.266125090" watchObservedRunningTime="2026-10-30 19:46:35.13781884 +0000 UTC m=+719.267052763" Oct 30 19:46:35.236430 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.236344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:35.351851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:35.351821 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:36.092487 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:36.092459 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" event={"ID":"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8","Type":"ContainerStarted","Data":"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c"} Oct 30 19:46:36.092890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:36.092744 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:36.164837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:36.164787 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" podStartSLOduration=1.866178683 podStartE2EDuration="2.164770776s" totalImagesPullingTime="298.592093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.164073914 +0000 UTC m=+720.293307836" watchObservedRunningTime="2026-10-30 19:46:36.164770776 +0000 UTC m=+720.294004699" Oct 30 19:46:36.206486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:36.206451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:36.318123 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:36.318088 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.129477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.128888 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.244993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.244958 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.402696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.402665 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 19:46:37.408202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.408180 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.470560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.470469 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-5db436\"" Oct 30 19:46:37.473762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.473734 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 19:46:37.484872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.484844 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-5db436-dockercfg-qxnmr\"" Oct 30 19:46:37.515551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.513126 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.515551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.513216 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rld5t\" (UniqueName: \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.614087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.614044 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.614278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.614116 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rld5t\" (UniqueName: \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.615080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.615054 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.636046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.636017 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rld5t\" (UniqueName: \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\") pod \"payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:37.774877 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:37.774784 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:38.032840 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.032764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.084672 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:38.084632 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfef63e_8821_451f_9403_fc3b7356f408.slice/crio-28d4048413eb93722aabc164debc2a10d05d6f42f01e23eac9261697828953c5 WatchSource:0}: Error finding container 28d4048413eb93722aabc164debc2a10d05d6f42f01e23eac9261697828953c5: Status 404 returned error can't find the container with id 28d4048413eb93722aabc164debc2a10d05d6f42f01e23eac9261697828953c5 Oct 30 19:46:38.085280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.085255 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 19:46:38.098827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.098803 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 19:46:38.104394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.104373 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.104919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.104892 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" event={"ID":"3cfef63e-8821-451f-9403-fc3b7356f408","Type":"ContainerStarted","Data":"28d4048413eb93722aabc164debc2a10d05d6f42f01e23eac9261697828953c5"} Oct 30 19:46:38.132534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.127748 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-6c4dd1-serving-cert\"" Oct 30 19:46:38.132534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.132025 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 19:46:38.173820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.173790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.220784 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.220749 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpzk4\" (UniqueName: \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.220911 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.220853 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.221019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.220936 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.221019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.220986 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.288687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.288574 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.325526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.324143 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpzk4\" (UniqueName: \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.325526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.324220 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.325526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.324385 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.325526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.324412 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.357519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.335374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.357519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.335452 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.357519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.338158 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.372518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.369318 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpzk4\" (UniqueName: \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\") pod \"maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.404109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.404044 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.462523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.459726 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:38.798579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:38.794856 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 19:46:38.804397 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:38.804368 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50891422_97f8_4a2a_a2e9_bae193d83192.slice/crio-f59cecbd104457807dcf0fc015ad3e7c2e28e9b904fc5e8338aebd21d9774dc8 WatchSource:0}: Error finding container f59cecbd104457807dcf0fc015ad3e7c2e28e9b904fc5e8338aebd21d9774dc8: Status 404 returned error can't find the container with id f59cecbd104457807dcf0fc015ad3e7c2e28e9b904fc5e8338aebd21d9774dc8 Oct 30 19:46:39.112075 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:39.111976 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" event={"ID":"50891422-97f8-4a2a-a2e9-bae193d83192","Type":"ContainerStarted","Data":"f59cecbd104457807dcf0fc015ad3e7c2e28e9b904fc5e8338aebd21d9774dc8"} Oct 30 19:46:39.113366 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:39.113341 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" event={"ID":"3cfef63e-8821-451f-9403-fc3b7356f408","Type":"ContainerStarted","Data":"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b"} Oct 30 19:46:39.113681 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:39.113663 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:39.180096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:39.166819 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" podStartSLOduration=2.166797158 podStartE2EDuration="2.166797158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:39.164899799 +0000 UTC m=+723.294133723" watchObservedRunningTime="2026-10-30 19:46:39.166797158 +0000 UTC m=+723.296031079" Oct 30 19:46:39.833140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:39.833059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:40.120184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:40.120087 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" event={"ID":"50891422-97f8-4a2a-a2e9-bae193d83192","Type":"ContainerStarted","Data":"ff37e44690340d8a1e8e9a598e82eb6dad87a03097f108e217b4e3def3bfb4fb"} Oct 30 19:46:40.120394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:40.120376 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:40.178519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:40.172613 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" podStartSLOduration=1.650224863 podStartE2EDuration="2.172594703s" totalImagesPullingTime="522.36984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:40.171792695 +0000 UTC m=+724.301026615" watchObservedRunningTime="2026-10-30 19:46:40.172594703 +0000 UTC m=+724.301828626" Oct 30 19:46:41.596073 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:41.595851 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:41.719605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:41.717156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:42.106776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:42.106751 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 19:46:43.695885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:43.695838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:44.081637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:44.081616 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 19:46:44.081777 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:44.081679 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 19:46:44.723159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:44.722560 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.090724 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:45.090634 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 19:46:45.302424 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:45.302386 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.422480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:45.422429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.565713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:45.565665 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.067439 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:46:46.067404 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:46:46.095527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.095472 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 19:46:46.101120 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.101092 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 19:46:46.101261 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.101178 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.125009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.124974 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 19:46:46.134806 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.134780 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-109cfe-serving-cert\"" Oct 30 19:46:46.135008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.134796 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 19:46:46.165087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.165055 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.165087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.165091 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.165327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.165134 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.165327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.165170 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5t2h\" (UniqueName: \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.193726 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.193693 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.265766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.265735 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.265954 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.265800 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.265954 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.265871 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5t2h\" (UniqueName: \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.266110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.266089 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.266344 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:46:46.266323 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-serving-cert: secret "maas-api-e2e-worker-w1-109cfe-serving-cert" not found Oct 30 19:46:46.266463 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:46:46.266407 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls podName:527f1372-f22d-450b-bca2-09d79e750070 nodeName:}" failed. No retries permitted until 2026-10-30 19:46:46.766385952 +0000 UTC m=+730.895619853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls") pod "maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" (UID: "527f1372-f22d-450b-bca2-09d79e750070") : secret "maas-api-e2e-worker-w1-109cfe-serving-cert" not found Oct 30 19:46:46.266880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.266852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.269305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.269278 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.289294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.289254 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5t2h\" (UniqueName: \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.303223 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.303179 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.415677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.415644 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.772923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.772822 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:46.775636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:46.775607 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") pod \"maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:47.015320 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:47.015286 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:47.082893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:47.081675 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:47.402813 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:47.402782 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527f1372_f22d_450b_bca2_09d79e750070.slice/crio-30d2859139f40ed27844bfbd719294ec8df2bad7b51cc781ab53e50877a0f0eb WatchSource:0}: Error finding container 30d2859139f40ed27844bfbd719294ec8df2bad7b51cc781ab53e50877a0f0eb: Status 404 returned error can't find the container with id 30d2859139f40ed27844bfbd719294ec8df2bad7b51cc781ab53e50877a0f0eb Oct 30 19:46:47.403546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:47.403520 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 19:46:48.173173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.173142 4959 generic.go:356] "Generic (PLEG): container finished" podID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerID="a456e0a59c5d80f86d93626f4839293350ec3e6ec38a1317d1c4558349801c1f" exitCode=0 Oct 30 19:46:48.173739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.173222 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerDied","Data":"a456e0a59c5d80f86d93626f4839293350ec3e6ec38a1317d1c4558349801c1f"} Oct 30 19:46:48.173739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.173658 4959 scope.go:122] "RemoveContainer" containerID="a456e0a59c5d80f86d93626f4839293350ec3e6ec38a1317d1c4558349801c1f" Oct 30 19:46:48.175103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175068 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.175225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175138 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.175225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175169 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.175225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175195 4959 generic.go:356] "Generic (PLEG): container finished" podID="01d7759f-78c0-4427-8451-fe3022de4942" containerID="db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab" exitCode=0 Oct 30 19:46:48.175343 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175260 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerDied","Data":"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab"} Oct 30 19:46:48.175729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.175590 4959 scope.go:122] "RemoveContainer" containerID="db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab" Oct 30 19:46:48.176884 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.176827 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.177005 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.176916 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.177005 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.176968 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:46:48.177391 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.177308 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" event={"ID":"527f1372-f22d-450b-bca2-09d79e750070","Type":"ContainerStarted","Data":"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7"} Oct 30 19:46:48.177391 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.177353 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" event={"ID":"527f1372-f22d-450b-bca2-09d79e750070","Type":"ContainerStarted","Data":"30d2859139f40ed27844bfbd719294ec8df2bad7b51cc781ab53e50877a0f0eb"} Oct 30 19:46:48.177813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.177476 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:48.354831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.353302 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" podStartSLOduration=2.002001196 podStartE2EDuration="2.353282353s" totalImagesPullingTime="351.281157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:48.299940508 +0000 UTC m=+732.429174432" watchObservedRunningTime="2026-10-30 19:46:48.353282353 +0000 UTC m=+732.482516280" Oct 30 19:46:48.866331 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.866290 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:48.985128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:48.985085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:49.121579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.115038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:49.205541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.203473 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerStarted","Data":"5aacc19eaa3fa11a24274e5ecdd9d9a28763a4c20e078781f2e5a790523b9923"} Oct 30 19:46:49.213530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.213346 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerStarted","Data":"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762"} Oct 30 19:46:49.410530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.410065 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:49.410530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.410196 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:49.410530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.410240 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:49.555523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.551856 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:49.555523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.551995 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:49.555523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.552040 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:49.833295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:49.833209 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.153523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.143815 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 19:46:50.406571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.406287 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.406571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.406347 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:50.480422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.480386 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.551874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.551838 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.552798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.552766 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:50.596519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.596343 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.715406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.715284 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.847213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:50.847154 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:51.407024 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:51.406990 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.407692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:51.407645 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:51.557518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:51.556635 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.557518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:51.556714 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:52.408521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.407925 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.408521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.407985 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:52.597531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.585040 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.597531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.585098 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:52.805540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.805425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.933419 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.932826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.947414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:52.947380 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 19:46:53.155520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.152632 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 19:46:53.155520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.152848 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.168763 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.168734 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:53.185234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.185201 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8a3426-serving-cert\"" Oct 30 19:46:53.257528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.256427 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.289532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.288384 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmw9x\" (UniqueName: \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.289532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.288633 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.289532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.288709 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.397197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.397147 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmw9x\" (UniqueName: \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.397885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.397854 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.398127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.398111 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.398325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.398311 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.399278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.399251 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.416226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.416125 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.416821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.416768 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:53.442611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.432447 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmw9x\" (UniqueName: \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.454534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.453156 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.474527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.471068 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:53.548754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.548720 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.549027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.548995 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:53.589506 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:53.589462 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:54.360581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.359683 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 19:46:54.447611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.447579 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 30 19:46:54.448293 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.448164 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 30 19:46:54.550586 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.550553 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:46:54.550842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.550813 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 19:46:54.609556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.609520 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 19:46:54.696432 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:54.696395 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:54.735195 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:46:54.735151 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0157790_cbb5_4d18_8946_a6af70e6d6a2.slice/crio-12c4b0bd719631e8d6d48f2fe343f672b0c847f4f2a9da9cb3c503c954341a36 WatchSource:0}: Error finding container 12c4b0bd719631e8d6d48f2fe343f672b0c847f4f2a9da9cb3c503c954341a36: Status 404 returned error can't find the container with id 12c4b0bd719631e8d6d48f2fe343f672b0c847f4f2a9da9cb3c503c954341a36 Oct 30 19:46:55.410520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.410448 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:55.410930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.410869 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:55.411956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.411900 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 19:46:55.698785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.698637 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" event={"ID":"c0157790-cbb5-4d18-8946-a6af70e6d6a2","Type":"ContainerStarted","Data":"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239"} Oct 30 19:46:55.698785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.698692 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" event={"ID":"c0157790-cbb5-4d18-8946-a6af70e6d6a2","Type":"ContainerStarted","Data":"12c4b0bd719631e8d6d48f2fe343f672b0c847f4f2a9da9cb3c503c954341a36"} Oct 30 19:46:55.698785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.698737 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:46:55.989477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.988155 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:55.999190 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.999118 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:55.999404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:55.999212 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 19:46:56.039672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:56.039620 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" podStartSLOduration=3.63007025 podStartE2EDuration="4.039604294s" totalImagesPullingTime="409.534044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:55.755205467 +0000 UTC m=+739.884439390" watchObservedRunningTime="2026-10-30 19:46:56.039604294 +0000 UTC m=+740.168838215" Oct 30 19:46:56.445643 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:56.445607 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.533178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:58.533143 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:59.625829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:59.625771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:59.711221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:59.710809 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:47322->10.132.0.43:15021: read: connection reset by peer" start-of-body= Oct 30 19:46:59.711221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:46:59.710890 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:47322->10.132.0.43:15021: read: connection reset by peer" Oct 30 19:47:00.080215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.080183 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.085645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.085622 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:47:00.085765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.085688 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:47:00.199306 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.199265 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.420979 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.420931 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.726904 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.726830 4959 generic.go:356] "Generic (PLEG): container finished" podID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerID="0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34" exitCode=0 Oct 30 19:47:00.727257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.726915 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerDied","Data":"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34"} Oct 30 19:47:00.727409 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.727391 4959 scope.go:122] "RemoveContainer" containerID="0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34" Oct 30 19:47:00.728791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.728764 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:00.728906 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.728844 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:00.728906 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.728896 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:00.827070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.827041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.955639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:00.955453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.068232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:01.068148 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.711021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:01.710992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.743790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:01.743748 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerStarted","Data":"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde"} Oct 30 19:47:01.864530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:01.861827 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.046883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.046797 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.583939 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.583909 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.706011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.705984 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:47:02.706335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.706313 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:47:02.706389 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.706366 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:47:02.713630 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.713612 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 19:47:02.773216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.773176 4959 generic.go:356] "Generic (PLEG): container finished" podID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerID="db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9" exitCode=0 Oct 30 19:47:02.774028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.774003 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerDied","Data":"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9"} Oct 30 19:47:02.774922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.774384 4959 scope.go:122] "RemoveContainer" containerID="db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9" Oct 30 19:47:02.777434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.777175 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:02.777434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.777261 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:02.777434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:02.777312 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:03.706315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:03.706281 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:47:03.706532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:03.706336 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:47:03.798488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:03.798453 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerStarted","Data":"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af"} Oct 30 19:47:04.198522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.179901 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:47:04.198522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.180034 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:47:04.198522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.180080 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 19:47:04.364562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.364528 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:04.706753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.706712 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:47:04.706967 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:04.706772 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:47:05.179961 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:05.179924 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:47:05.180594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:05.180562 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 19:47:05.710521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:05.709977 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:47:05.710521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:05.710040 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:47:06.191538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.180066 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:47:06.191538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.180142 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 19:47:06.307298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.307261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:06.433324 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.433295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:06.689927 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.689851 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:06.706317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.706278 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:47:06.706523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.706340 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 30 19:47:06.819902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:06.819867 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.021598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.021484 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.179943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.179907 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:47:07.180231 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.180203 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 19:47:07.195793 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.195738 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.481356 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.481322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.670037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.669997 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.716450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.716420 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:47:07.717137 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.717119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:47:07.717919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:07.717901 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 19:47:08.180562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:08.180525 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:47:08.180743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:08.180584 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Oct 30 19:47:08.890359 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:08.890328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:09.318314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:09.318198 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:47:09.318687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:09.318659 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:47:09.319998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:09.319978 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 19:47:10.527284 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:10.527254 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:11.983678 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:11.983628 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.587050 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:12.587017 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.721813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:12.721761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.844229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:12.844125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:14.176032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:14.176001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:14.516983 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:14.516904 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:16.067856 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:47:16.067818 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:47:16.098689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:16.098662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:16.481693 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:16.481661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.465777 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.465742 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 19:47:17.471297 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.471279 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.506190 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.506161 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"kube-root-ca.crt\"" Oct 30 19:47:17.506370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.506185 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"openshift-service-ca.crt\"" Oct 30 19:47:17.506626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.506610 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"premium-w6-f349a1-kserve-self-signed-certs\"" Oct 30 19:47:17.508307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.508285 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 19:47:17.509883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.509862 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"default-dockercfg-9dr8w\"" Oct 30 19:47:17.532953 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.532921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.620304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620254 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.620304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbds\" (UniqueName: \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.620558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620431 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.620558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620450 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.620558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620481 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.620663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.620612 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.721977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.721891 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.721977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.721942 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.721987 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722006 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbds\" (UniqueName: \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722073 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722095 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722391 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722314 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722389 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.722450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.722417 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.724595 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.724567 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.724710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.724685 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.749409 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.749376 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbds\" (UniqueName: \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\") pod \"premium-w6-f349a1-kserve-8585c86b7d-6572f\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.782803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.782767 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:17.879103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:17.879064 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.090706 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:47:18.090679 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9df96ea_61ae_44c6_8f37_e8fab797b53c.slice/crio-ee22f54c810011154a797ff88bf2c68eae50e3b1d56d6941236b470723ace608 WatchSource:0}: Error finding container ee22f54c810011154a797ff88bf2c68eae50e3b1d56d6941236b470723ace608: Status 404 returned error can't find the container with id ee22f54c810011154a797ff88bf2c68eae50e3b1d56d6941236b470723ace608 Oct 30 19:47:18.091291 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:18.091268 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 19:47:18.928841 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:18.928797 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" event={"ID":"d9df96ea-61ae-44c6-8f37-e8fab797b53c","Type":"ContainerStarted","Data":"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c"} Oct 30 19:47:18.929369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:18.928851 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" event={"ID":"d9df96ea-61ae-44c6-8f37-e8fab797b53c","Type":"ContainerStarted","Data":"ee22f54c810011154a797ff88bf2c68eae50e3b1d56d6941236b470723ace608"} Oct 30 19:47:18.975837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:18.975771 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podStartSLOduration=1.9757507319999998 podStartE2EDuration="1.975750732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:18.973653783 +0000 UTC m=+763.102887708" watchObservedRunningTime="2026-10-30 19:47:18.975750732 +0000 UTC m=+763.104984654" Oct 30 19:47:19.582993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:19.582960 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:19.783592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:19.783559 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:21.177468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.177434 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:21.958241 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.958211 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 19:47:21.962137 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.962114 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:21.992886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.992857 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"openshift-service-ca.crt\"" Oct 30 19:47:21.992886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.992859 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"default-dockercfg-7v28s\"" Oct 30 19:47:21.992886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.992892 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"simulator-w1-109cfe-kserve-self-signed-certs\"" Oct 30 19:47:21.993184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.992935 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"kube-root-ca.crt\"" Oct 30 19:47:21.997916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:21.997886 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 19:47:22.022175 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.022146 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.068064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.068218 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068067 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72wr\" (UniqueName: \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.068218 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068169 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.068218 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068215 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.068346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068301 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.068386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.068373 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169152 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169200 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169252 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169283 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169311 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72wr\" (UniqueName: \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169367 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169689 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169712 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.169768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.169756 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.171823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.171804 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.172071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.172056 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.218470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.218379 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72wr\" (UniqueName: \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\") pod \"simulator-w1-109cfe-kserve-56c9fc549b-wpbng\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.276561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.276527 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:22.581748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.581723 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 19:47:22.581865 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:47:22.581772 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod857c2303_7886_445b_8fe9_d1326b68e879.slice/crio-b99cd97bef5922fa60492a6dc9916e2e82ff623872bdc72826287d73a721675c WatchSource:0}: Error finding container b99cd97bef5922fa60492a6dc9916e2e82ff623872bdc72826287d73a721675c: Status 404 returned error can't find the container with id b99cd97bef5922fa60492a6dc9916e2e82ff623872bdc72826287d73a721675c Oct 30 19:47:22.827205 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.827173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:22.950440 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.950408 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" event={"ID":"857c2303-7886-445b-8fe9-d1326b68e879","Type":"ContainerStarted","Data":"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667"} Oct 30 19:47:22.950440 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.950447 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" event={"ID":"857c2303-7886-445b-8fe9-d1326b68e879","Type":"ContainerStarted","Data":"b99cd97bef5922fa60492a6dc9916e2e82ff623872bdc72826287d73a721675c"} Oct 30 19:47:22.999044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:22.998997 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podStartSLOduration=1.9989828840000001 podStartE2EDuration="1.998982884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:22.998100412 +0000 UTC m=+767.127334344" watchObservedRunningTime="2026-10-30 19:47:22.998982884 +0000 UTC m=+767.128216804" Oct 30 19:47:23.276988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:23.276913 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:24.325211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:24.325174 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.712181 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:24.712150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.377416 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.377383 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.812798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.812761 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 19:47:26.817941 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.817924 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.834814 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.834791 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6c4dd1-openshift-default-dockercfg-tw2l4\"" Oct 30 19:47:26.849786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.849759 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 19:47:26.928437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928395 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928445 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928537 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928604 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928637 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928697 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcgd\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928756 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.928951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928909 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:26.929049 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:26.928952 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.024845 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.024807 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:27.030097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030073 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030117 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030138 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030163 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030190 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030235 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcgd\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030266 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030573 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030609 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030652 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.030936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.030897 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.031131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.031110 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.031400 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.031297 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.031461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.031416 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.032954 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.032934 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.033276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.033251 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.058076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.058034 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcgd\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.058711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.058695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.131162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.131084 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:27.483054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.483025 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 19:47:27.483453 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:47:27.483430 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e682b0_f4bd_403b_b358_73e30d2d6b3e.slice/crio-ff0cab8426a44c8fd3c7ee40f3221324a489341a8aa173e6ff28cebc1dbfd285 WatchSource:0}: Error finding container ff0cab8426a44c8fd3c7ee40f3221324a489341a8aa173e6ff28cebc1dbfd285: Status 404 returned error can't find the container with id ff0cab8426a44c8fd3c7ee40f3221324a489341a8aa173e6ff28cebc1dbfd285 Oct 30 19:47:27.485869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.485834 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:27.485994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.485910 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:27.485994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.485954 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:47:27.783687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.783606 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:27.789828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.789800 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:27.799616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.799596 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:27.979097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.979057 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" event={"ID":"15e682b0-f4bd-403b-b358-73e30d2d6b3e","Type":"ContainerStarted","Data":"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510"} Oct 30 19:47:27.979260 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.979121 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" event={"ID":"15e682b0-f4bd-403b-b358-73e30d2d6b3e","Type":"ContainerStarted","Data":"ff0cab8426a44c8fd3c7ee40f3221324a489341a8aa173e6ff28cebc1dbfd285"} Oct 30 19:47:27.994951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:27.994925 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:47:28.028634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:28.028588 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podStartSLOduration=2.028574955 podStartE2EDuration="2.028574955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:28.026870182 +0000 UTC m=+772.156104102" watchObservedRunningTime="2026-10-30 19:47:28.028574955 +0000 UTC m=+772.157808875" Oct 30 19:47:28.083543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:28.083420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.131672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:28.131641 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:28.133292 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:28.133267 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:28.133412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:28.133322 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:29.132302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:29.132263 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:29.132862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:29.132335 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:29.867520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:29.867159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.137573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:30.132032 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:30.137573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:30.132117 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:31.132056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:31.132025 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:31.132230 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:31.132080 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:31.573686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:31.573650 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:32.131469 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:32.131439 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:32.131686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:32.131511 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:32.277274 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:32.277241 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:32.290301 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:32.290269 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:33.034250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:33.034224 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:47:33.132060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:33.132032 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:47:33.132227 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:33.132093 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 19:47:33.183224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:33.183192 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.136597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:34.136567 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:34.137016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:34.136813 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:34.137467 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:34.137452 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 19:47:34.741874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:34.741844 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:36.249265 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:36.249230 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:37.812343 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:37.812249 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.133289 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:39.133233 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.739863 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:39.739828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:41.066094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:41.066063 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:41.416337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:41.416303 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.027951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:43.027922 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.724057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:44.724022 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:46.065091 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:47:46.065054 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:47:46.338537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:46.338420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:48.439519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:48.438652 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:50.214107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:50.214069 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.861401 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:51.861345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.511028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.511001 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 19:47:52.514908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.514889 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.537516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.537478 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"kube-root-ca.crt\"" Oct 30 19:47:52.537625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.537543 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"openshift-service-ca.crt\"" Oct 30 19:47:52.550907 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.550883 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"premium-w3-01f7d3-kserve-self-signed-certs\"" Oct 30 19:47:52.551017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.550995 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"default-dockercfg-8cntf\"" Oct 30 19:47:52.557328 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.557308 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 19:47:52.573615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.573571 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.593003 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.592981 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.593140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.593025 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.593140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.593051 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv94l\" (UniqueName: \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.593262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.593199 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.593326 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.593308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.593369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.593358 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.694620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694587 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.694819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694635 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv94l\" (UniqueName: \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.694819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694677 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.694819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694738 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.694819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694788 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.695033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.694873 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.695171 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.695143 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.695262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.695158 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.695262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.695244 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.697025 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.697007 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.697175 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.697159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.746004 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.745970 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv94l\" (UniqueName: \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\") pod \"premium-w3-01f7d3-kserve-57b97b98cb-qpq5w\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:52.826227 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:52.826137 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:53.130337 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:47:53.130309 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b62495_d391_4c86_b5c6_0f5989657f27.slice/crio-f64ce8871c6dc821d381cae9652c77bb85ca65e1ac4375e620183a42890ade59 WatchSource:0}: Error finding container f64ce8871c6dc821d381cae9652c77bb85ca65e1ac4375e620183a42890ade59: Status 404 returned error can't find the container with id f64ce8871c6dc821d381cae9652c77bb85ca65e1ac4375e620183a42890ade59 Oct 30 19:47:53.130686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:53.130479 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 19:47:53.157302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:53.157264 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" event={"ID":"36b62495-d391-4c86-b5c6-0f5989657f27","Type":"ContainerStarted","Data":"f64ce8871c6dc821d381cae9652c77bb85ca65e1ac4375e620183a42890ade59"} Oct 30 19:47:53.486935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:53.486893 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.067003 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.066967 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 19:47:54.070813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.070790 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.092901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.092875 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 19:47:54.103776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.103754 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"premium-w1-109cfe-kserve-self-signed-certs\"" Oct 30 19:47:54.108193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108166 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.108322 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108291 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.108393 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108341 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.108468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108445 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.108568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108531 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.108629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.108588 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq48d\" (UniqueName: \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.116530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.116508 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.164314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.164285 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" event={"ID":"36b62495-d391-4c86-b5c6-0f5989657f27","Type":"ContainerStarted","Data":"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243"} Oct 30 19:47:54.209396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.209528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209425 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.209528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209461 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.209646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209531 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq48d\" (UniqueName: \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.209646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209616 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.209754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.209697 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.210291 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.210265 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.210383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.210295 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.210383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.210337 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.212705 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.212684 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.212899 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.212879 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.214992 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.214945 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podStartSLOduration=2.214929123 podStartE2EDuration="2.214929123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:54.211814673 +0000 UTC m=+798.341048590" watchObservedRunningTime="2026-10-30 19:47:54.214929123 +0000 UTC m=+798.344163046" Oct 30 19:47:54.233407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.233382 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq48d\" (UniqueName: \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\") pod \"premium-w1-109cfe-kserve-64b64f4f85-hp6g9\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.387209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.387174 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:54.715933 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:47:54.715906 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda4db64c_00b4_417d_9bb3_d83c466c79be.slice/crio-ff728ce803b01f45bff90ae4193860a80e1ea43d1bb34eb38911547b1063dc8b WatchSource:0}: Error finding container ff728ce803b01f45bff90ae4193860a80e1ea43d1bb34eb38911547b1063dc8b: Status 404 returned error can't find the container with id ff728ce803b01f45bff90ae4193860a80e1ea43d1bb34eb38911547b1063dc8b Oct 30 19:47:54.716249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.716225 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 19:47:54.826569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:54.826543 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:47:55.082872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:55.082791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:55.171297 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:55.171255 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" event={"ID":"da4db64c-00b4-417d-9bb3-d83c466c79be","Type":"ContainerStarted","Data":"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f"} Oct 30 19:47:55.171297 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:55.171296 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" event={"ID":"da4db64c-00b4-417d-9bb3-d83c466c79be","Type":"ContainerStarted","Data":"ff728ce803b01f45bff90ae4193860a80e1ea43d1bb34eb38911547b1063dc8b"} Oct 30 19:47:55.219330 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:55.219269 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podStartSLOduration=1.219250656 podStartE2EDuration="1.219250656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:55.216746091 +0000 UTC m=+799.345980014" watchObservedRunningTime="2026-10-30 19:47:55.219250656 +0000 UTC m=+799.348484579" Oct 30 19:47:55.387794 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:55.387764 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:47:56.854786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:56.854750 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.657203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:47:58.657172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:00.511240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:00.511202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.670520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:02.666880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.831518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:02.830693 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:48:02.850993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:02.850960 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:48:03.231521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:03.231482 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:48:04.356538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:04.356490 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:04.387763 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:04.387732 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:48:04.400527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:04.400485 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:48:05.010641 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:05.010610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:05.242042 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:05.242015 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:48:06.702334 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:06.702298 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:06.934106 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:06.934069 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:08.301735 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:08.301707 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:09.902629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:09.902592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.391233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:11.391197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.774121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:11.774040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:13.632454 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:13.632423 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:15.329304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:15.329269 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:16.075179 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:48:16.075145 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:48:17.353620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:17.353586 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:19.084120 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:19.084077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.920685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:20.920643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.712460 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:22.712429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:24.270759 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:24.270729 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:25.876544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:25.876512 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:27.517765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:27.517732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:27.909991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:27.909956 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:29.209249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:29.209216 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:30.803165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:30.803134 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:32.361667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:32.361639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:33.913714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:33.913678 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:35.630111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:35.630073 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:37.311044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:37.311011 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.282729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:39.282687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:41.084371 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:41.084338 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:42.797552 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:42.797510 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:44.407041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:44.407006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.071048 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:48:46.071002 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:48:46.219118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:46.219086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:47.946097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:47.946067 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:49.597623 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:49.597587 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:51.094278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:51.094232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:52.744530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:52.744481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:54.493204 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:54.493170 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:55.990478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:55.990438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:57.688896 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:57.688858 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:59.509439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:48:59.509404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:01.203302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:01.203268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.144481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:03.144447 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:04.848772 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:04.848743 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.407368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:06.407336 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:08.196602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:08.196569 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:09.969811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:09.969777 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.079926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.079887 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.156688 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.156653 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 19:49:12.161445 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.161422 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.194042 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.194009 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 19:49:12.254292 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254244 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254311 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254398 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhbz\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254516 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254551 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254584 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254624 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254652 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.254988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.254727 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355687 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355734 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355797 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355831 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355856 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhbz\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355921 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.355977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355953 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.355982 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356124 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356277 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356323 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356379 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356715 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.356868 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.356847 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.358638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.358610 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.358779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.358762 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.377395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.377364 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.377613 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.377474 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhbz\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.474937 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.474906 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:12.796855 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:12.796827 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd37d968_606c_4651_8bb8_fbc56826f361.slice/crio-225c4274aa0629fbd6a606d4e62cb7dff327df7f2b6ba2360161e888bdce2170 WatchSource:0}: Error finding container 225c4274aa0629fbd6a606d4e62cb7dff327df7f2b6ba2360161e888bdce2170: Status 404 returned error can't find the container with id 225c4274aa0629fbd6a606d4e62cb7dff327df7f2b6ba2360161e888bdce2170 Oct 30 19:49:12.797884 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.797855 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 19:49:12.799593 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.799561 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:12.799680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.799626 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:12.799680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:12.799656 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:13.700439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:13.700398 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.705027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:13.704999 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" event={"ID":"dd37d968-606c-4651-8bb8-fbc56826f361","Type":"ContainerStarted","Data":"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4"} Oct 30 19:49:13.705150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:13.705064 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" event={"ID":"dd37d968-606c-4651-8bb8-fbc56826f361","Type":"ContainerStarted","Data":"225c4274aa0629fbd6a606d4e62cb7dff327df7f2b6ba2360161e888bdce2170"} Oct 30 19:49:13.779533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:13.778334 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podStartSLOduration=1.7783066 podStartE2EDuration="1.7783066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:13.775082495 +0000 UTC m=+877.904316421" watchObservedRunningTime="2026-10-30 19:49:13.7783066 +0000 UTC m=+877.907540573" Oct 30 19:49:14.475962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:14.475935 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:14.477950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:14.477923 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:14.478122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:14.477980 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:15.393693 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:15.393655 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.484521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:15.483868 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:15.484521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:15.483928 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:16.084875 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:49:16.084838 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:49:16.475866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:16.475833 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:16.476267 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:16.475886 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:17.109749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:17.109714 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.475957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:17.475924 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:17.476392 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:17.475976 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:18.476053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:18.476022 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:18.476569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:18.476079 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:18.957959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:18.957768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.475784 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.475736 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:49:19.475982 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.475803 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 30 19:49:19.604766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.604720 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 19:49:19.609933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.609906 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.629163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.629136 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 19:49:19.639340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.639312 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"kube-root-ca.crt\"" Oct 30 19:49:19.639340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.639312 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"openshift-service-ca.crt\"" Oct 30 19:49:19.639566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.639389 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"default-dockercfg-jb4r7\"" Oct 30 19:49:19.639566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.639435 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5db436\"/\"distinct-w0-5db436-kserve-self-signed-certs\"" Oct 30 19:49:19.657716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.657683 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.677188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677149 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.677381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677293 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwmz\" (UniqueName: \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.677451 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677409 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.677514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677462 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.677668 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677644 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.677809 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.677786 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.779313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.779220 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.779655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.779630 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwmz\" (UniqueName: \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.779872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.779849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.780002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.779910 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.780518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.780105 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.780518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.780391 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.790677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.781101 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.790677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.781419 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.790677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.781608 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.816532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.815603 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.829521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.821865 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.829705 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.829560 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwmz\" (UniqueName: \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\") pod \"distinct-w0-5db436-kserve-768cc9c8dc-ljv6m\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:19.925767 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:19.925733 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:20.241817 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:20.241787 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d1c6a4_47e9_4ecb_8597_63602b53e3b9.slice/crio-ead1d8c87e57375257046e63ad4a03489d13f8b04288b3181a0823c0bf0b2190 WatchSource:0}: Error finding container ead1d8c87e57375257046e63ad4a03489d13f8b04288b3181a0823c0bf0b2190: Status 404 returned error can't find the container with id ead1d8c87e57375257046e63ad4a03489d13f8b04288b3181a0823c0bf0b2190 Oct 30 19:49:20.241817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.241815 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 19:49:20.480089 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.480060 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:20.480432 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.480413 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:20.481776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.481750 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 19:49:20.690437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.690406 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:20.745670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.745625 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" event={"ID":"17d1c6a4-47e9-4ecb-8597-63602b53e3b9","Type":"ContainerStarted","Data":"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96"} Oct 30 19:49:20.745670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.745677 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" event={"ID":"17d1c6a4-47e9-4ecb-8597-63602b53e3b9","Type":"ContainerStarted","Data":"ead1d8c87e57375257046e63ad4a03489d13f8b04288b3181a0823c0bf0b2190"} Oct 30 19:49:20.789114 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.789067 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podStartSLOduration=1.789053527 podStartE2EDuration="1.789053527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:20.787701579 +0000 UTC m=+884.916935512" watchObservedRunningTime="2026-10-30 19:49:20.789053527 +0000 UTC m=+884.918287448" Oct 30 19:49:20.932519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:20.931131 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:21.751655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:21.751622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:22.375296 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:22.375261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:24.071274 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:24.071243 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:25.698001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:25.697947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:27.281874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.281816 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 19:49:27.291522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.288603 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.321565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.321466 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5db436-openshift-default-dockercfg-n7492\"" Oct 30 19:49:27.325234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.325195 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405224 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405283 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405341 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405375 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405410 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.405521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405441 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjtf\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.406016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405534 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.406016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405588 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.406016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.405636 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.506898 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjtf\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507005 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507059 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507109 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507220 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507255 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.507544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507315 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.508042 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.507793 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.508473 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.508550 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.508889 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.509464 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.509747 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.512848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.510479 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.517579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.517210 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.517748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.517667 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.529597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.529530 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjtf\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.533566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.533342 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.616967 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.616927 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:27.740922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.740890 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 19:49:27.745816 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.745793 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.764598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.764554 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a3426-openshift-default-dockercfg-4xlnw\"" Oct 30 19:49:27.774521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.774474 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 19:49:27.811936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.811784 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.811936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.811823 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.811937 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.811976 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.812067 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dhl\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.812181 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812277 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.812221 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812372 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.812356 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.812479 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.812463 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.913328 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913272 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.913486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913338 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.913486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913388 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dhl\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.913486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913426 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.913638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913612 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913732 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913822 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913939 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913964 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.913986 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.914328 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.914401 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.914479 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.914804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.914780 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.916332 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.916269 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.917140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.917120 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.937428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.937391 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.937967 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.937941 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dhl\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:27.954255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.954228 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 19:49:27.955106 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:27.955070 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded746e9e_a437_4203_98a7_15c919925e70.slice/crio-4114e66d9dc5549b474ba09038476889bf3d53f916560f484f8ddf727d436355 WatchSource:0}: Error finding container 4114e66d9dc5549b474ba09038476889bf3d53f916560f484f8ddf727d436355: Status 404 returned error can't find the container with id 4114e66d9dc5549b474ba09038476889bf3d53f916560f484f8ddf727d436355 Oct 30 19:49:27.958014 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.957978 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:27.958089 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.958051 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:27.958089 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:27.958077 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:28.004450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.004420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.081572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.077980 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:28.125360 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.125328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.415225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.415196 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 19:49:28.416926 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:28.416894 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663da9c7_6fc8_4c35_b9b1_cc77d2fecdcd.slice/crio-5d80f0194a90fe6ccfb6f17a20cdcab106916e72e23ab13141dce32dcf5f9738 WatchSource:0}: Error finding container 5d80f0194a90fe6ccfb6f17a20cdcab106916e72e23ab13141dce32dcf5f9738: Status 404 returned error can't find the container with id 5d80f0194a90fe6ccfb6f17a20cdcab106916e72e23ab13141dce32dcf5f9738 Oct 30 19:49:28.420185 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.420142 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:28.420294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.420216 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:28.420294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.420254 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:49:28.876946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.876843 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" event={"ID":"ed746e9e-a437-4203-98a7-15c919925e70","Type":"ContainerStarted","Data":"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd"} Oct 30 19:49:28.876946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.876898 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" event={"ID":"ed746e9e-a437-4203-98a7-15c919925e70","Type":"ContainerStarted","Data":"4114e66d9dc5549b474ba09038476889bf3d53f916560f484f8ddf727d436355"} Oct 30 19:49:28.878663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.878638 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" event={"ID":"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd","Type":"ContainerStarted","Data":"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537"} Oct 30 19:49:28.878795 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.878678 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" event={"ID":"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd","Type":"ContainerStarted","Data":"5d80f0194a90fe6ccfb6f17a20cdcab106916e72e23ab13141dce32dcf5f9738"} Oct 30 19:49:28.926662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.926601 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podStartSLOduration=1.926583914 podStartE2EDuration="1.926583914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:28.924819609 +0000 UTC m=+893.054053544" watchObservedRunningTime="2026-10-30 19:49:28.926583914 +0000 UTC m=+893.055817835" Oct 30 19:49:28.975178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:28.975103 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podStartSLOduration=1.975083036 podStartE2EDuration="1.975083036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:28.973560819 +0000 UTC m=+893.102794743" watchObservedRunningTime="2026-10-30 19:49:28.975083036 +0000 UTC m=+893.104316958" Oct 30 19:49:29.078811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.078765 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:29.080638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.080606 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:29.080801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.080670 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:29.617643 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.617560 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:29.619154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.619123 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:29.619288 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.619176 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:29.697520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.689351 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:29.926503 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.926459 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:29.939105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:29.939080 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:30.090521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:30.088905 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:30.090521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:30.088964 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:30.618171 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:30.618132 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:30.618606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:30.618184 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:30.920753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:30.920720 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:49:31.087031 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:31.086990 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:31.087311 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:31.087283 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:31.447676 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:31.447638 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:31.618521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:31.617899 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:31.618521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:31.617958 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:32.081638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:32.081603 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:32.081879 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:32.081853 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:32.618235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:32.618185 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:32.618443 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:32.618248 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:33.084892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:33.083970 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:33.084892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:33.084025 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:33.084892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:33.084825 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:33.617988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:33.617959 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:33.618203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:33.618177 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:34.083520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:34.082607 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:34.083520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:34.082665 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:34.617845 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:34.617809 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:49:34.618318 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:34.617870 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:49:34.785760 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:34.785720 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:35.079127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:35.079083 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:49:35.079319 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:35.079157 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 19:49:35.622149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:35.622122 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:35.622647 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:35.622593 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:35.623858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:35.623836 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 19:49:36.067797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.067772 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:49:36.068080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.068056 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:49:36.086108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.086079 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:36.086790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.086568 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:36.087681 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.087659 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 19:49:36.312066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.312032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.404573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.404529 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 19:49:36.410697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.410674 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.426206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.426153 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 19:49:36.426462 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.426443 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-109cfe\"/\"distinct-w1-109cfe-kserve-self-signed-certs\"" Oct 30 19:49:36.462273 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.462242 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.501251 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501219 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.501393 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501255 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55p6\" (UniqueName: \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.501393 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501291 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.501536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501432 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.501536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501480 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.501631 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.501564 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604233 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604379 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604423 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604549 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604594 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55p6\" (UniqueName: \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.604661 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.608159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.608444 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.609519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.608780 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.623281 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.623248 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.637918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.635211 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.643039 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.638519 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55p6\" (UniqueName: \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\") pod \"distinct-w1-109cfe-kserve-74b69d6f76-qlgmk\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:36.727234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:36.727145 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:37.028423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:37.028395 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 19:49:37.029305 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:37.029279 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5d4f89_f494_42e7_aa33_f37b64e8048c.slice/crio-78d5b2f5c564f5d6e7599c1bcdeb5c55637b37a93b19754aa8354d24c5e709a6 WatchSource:0}: Error finding container 78d5b2f5c564f5d6e7599c1bcdeb5c55637b37a93b19754aa8354d24c5e709a6: Status 404 returned error can't find the container with id 78d5b2f5c564f5d6e7599c1bcdeb5c55637b37a93b19754aa8354d24c5e709a6 Oct 30 19:49:37.908769 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:37.908735 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:38.001516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:38.001469 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" event={"ID":"4d5d4f89-f494-42e7-aa33-f37b64e8048c","Type":"ContainerStarted","Data":"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c"} Oct 30 19:49:38.001684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:38.001529 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" event={"ID":"4d5d4f89-f494-42e7-aa33-f37b64e8048c","Type":"ContainerStarted","Data":"78d5b2f5c564f5d6e7599c1bcdeb5c55637b37a93b19754aa8354d24c5e709a6"} Oct 30 19:49:38.043188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:38.043113 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podStartSLOduration=2.043096555 podStartE2EDuration="2.043096555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:38.04247999 +0000 UTC m=+902.171713911" watchObservedRunningTime="2026-10-30 19:49:38.043096555 +0000 UTC m=+902.172330474" Oct 30 19:49:38.727251 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:38.727223 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:39.398406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:39.398366 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.559988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:40.559954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:41.207457 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:41.207418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.072030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:43.071995 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.048414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:45.048379 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.356178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:45.356091 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.473041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:45.473008 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.064805 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:49:46.064776 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:49:46.727686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:46.727657 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:46.740717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:46.740692 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:47.030515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:47.030425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.135247 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:47.135218 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:49:49.051123 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:49.051089 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.111369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:51.111336 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.693250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:52.693218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.224621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.224586 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 19:49:54.228876 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.228852 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.250587 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.250556 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 19:49:54.260976 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.260958 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"distinct-w6-f349a1-kserve-self-signed-certs\"" Oct 30 19:49:54.304626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.304600 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.380376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380345 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.380376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380376 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgz7\" (UniqueName: \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.380658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380419 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.380658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380575 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.380658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380646 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.380827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.380699 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.454519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.444769 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482228 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482286 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgz7\" (UniqueName: \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482428 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482517 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.482547 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.483429 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.486560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.483712 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.487398 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.487374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.491102 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.491074 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.492916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.492895 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.507987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.507962 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgz7\" (UniqueName: \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\") pod \"distinct-w6-f349a1-kserve-cc965856f-f6rsw\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.543798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.543775 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:54.854089 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:54.854060 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 19:49:54.854199 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:49:54.854125 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be5f650_8675_4ddf_9317_ec8a8cf1c780.slice/crio-ca9d5d1c49fe2bfd591dd048dbe6cdb0ba2b5430270cee11e884d0b05d6004a2 WatchSource:0}: Error finding container ca9d5d1c49fe2bfd591dd048dbe6cdb0ba2b5430270cee11e884d0b05d6004a2: Status 404 returned error can't find the container with id ca9d5d1c49fe2bfd591dd048dbe6cdb0ba2b5430270cee11e884d0b05d6004a2 Oct 30 19:49:55.172909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:55.172882 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" event={"ID":"2be5f650-8675-4ddf-9317-ec8a8cf1c780","Type":"ContainerStarted","Data":"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17"} Oct 30 19:49:55.173083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:55.172917 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" event={"ID":"2be5f650-8675-4ddf-9317-ec8a8cf1c780","Type":"ContainerStarted","Data":"ca9d5d1c49fe2bfd591dd048dbe6cdb0ba2b5430270cee11e884d0b05d6004a2"} Oct 30 19:49:55.218647 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:55.218597 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podStartSLOduration=1.218582592 podStartE2EDuration="1.218582592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:55.216688896 +0000 UTC m=+919.345922816" watchObservedRunningTime="2026-10-30 19:49:55.218582592 +0000 UTC m=+919.347816546" Oct 30 19:49:55.544240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:55.544153 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:49:55.897676 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:55.897642 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:57.471707 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:57.471671 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.173521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:49:59.172606 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.411532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:01.411463 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:02.074712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.074602 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 19:50:02.080392 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.080364 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.098713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.098688 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 19:50:02.108711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.108672 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-01f7d3\"/\"distinct-w3-01f7d3-kserve-self-signed-certs\"" Oct 30 19:50:02.136631 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.136599 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:02.138224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138196 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.138363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138259 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.138433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138366 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhrf\" (UniqueName: \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.138537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138486 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.138603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138559 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.138650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.138608 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239482 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239447 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239533 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239579 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhrf\" (UniqueName: \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239671 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239697 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239743 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.239956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.239916 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.240042 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.240018 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.240302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.240280 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.242240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.242212 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.242842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.242819 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.287381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.287338 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhrf\" (UniqueName: \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\") pod \"distinct-w3-01f7d3-kserve-755b65cb59-dmg4b\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.396624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.396590 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:02.699689 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:02.699659 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod928c98ce_c4d5_417d_a715_e92d39072d88.slice/crio-149aee50fad615043edd9388cd064f98aaf6dde9e957010a1de4e74f2522f23d WatchSource:0}: Error finding container 149aee50fad615043edd9388cd064f98aaf6dde9e957010a1de4e74f2522f23d: Status 404 returned error can't find the container with id 149aee50fad615043edd9388cd064f98aaf6dde9e957010a1de4e74f2522f23d Oct 30 19:50:02.700111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:02.699899 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 19:50:03.011455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:03.011376 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.253697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:03.253664 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" event={"ID":"928c98ce-c4d5-417d-a715-e92d39072d88","Type":"ContainerStarted","Data":"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293"} Oct 30 19:50:03.253839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:03.253708 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" event={"ID":"928c98ce-c4d5-417d-a715-e92d39072d88","Type":"ContainerStarted","Data":"149aee50fad615043edd9388cd064f98aaf6dde9e957010a1de4e74f2522f23d"} Oct 30 19:50:03.306920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:03.306810 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podStartSLOduration=1.306792559 podStartE2EDuration="1.306792559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:03.306189434 +0000 UTC m=+927.435423367" watchObservedRunningTime="2026-10-30 19:50:03.306792559 +0000 UTC m=+927.436026483" Oct 30 19:50:03.397357 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:03.397324 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:04.544052 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:04.544020 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:50:04.557714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:04.557689 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:50:04.635162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:04.635117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.277470 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:05.277440 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:50:06.478138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:06.478104 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:06.733263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:06.733181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:08.120556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:08.113840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.784917 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:09.784886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.004619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.004590 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 19:50:11.008633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.008612 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.039922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.039896 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"distinct-w2-284bf6-kserve-self-signed-certs\"" Oct 30 19:50:11.040065 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.039934 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"kube-root-ca.crt\"" Oct 30 19:50:11.040065 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.040008 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"openshift-service-ca.crt\"" Oct 30 19:50:11.040065 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.040030 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-284bf6\"/\"default-dockercfg-ppjjf\"" Oct 30 19:50:11.042525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.042489 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 19:50:11.057135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.057109 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.076027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076007 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.076156 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076065 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrhv\" (UniqueName: \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.076222 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076182 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.076275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076248 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.076376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076357 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.076538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.076482 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209540 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209685 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrhv\" (UniqueName: \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209721 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209770 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209830 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.209920 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.210470 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.210746 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.213533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.211268 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.226760 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.226729 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.229285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.227167 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.246105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.246015 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrhv\" (UniqueName: \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\") pod \"distinct-w2-284bf6-kserve-c9cb575d7-9gblk\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.323338 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.323266 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:11.497002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.496973 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.634235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:11.634210 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 19:50:11.634482 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:11.634456 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e13789_931c_44a4_bbeb_60d874d39502.slice/crio-e5317e40a7920ed6503d178bf7e921f0a613bbc9cb1c88427f493d25c65506a7 WatchSource:0}: Error finding container e5317e40a7920ed6503d178bf7e921f0a613bbc9cb1c88427f493d25c65506a7: Status 404 returned error can't find the container with id e5317e40a7920ed6503d178bf7e921f0a613bbc9cb1c88427f493d25c65506a7 Oct 30 19:50:12.134276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.134240 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 19:50:12.138822 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.138800 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.165443 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.165394 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 19:50:12.310518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.310473 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" event={"ID":"33e13789-931c-44a4-bbeb-60d874d39502","Type":"ContainerStarted","Data":"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee"} Oct 30 19:50:12.310720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.310542 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" event={"ID":"33e13789-931c-44a4-bbeb-60d874d39502","Type":"ContainerStarted","Data":"e5317e40a7920ed6503d178bf7e921f0a613bbc9cb1c88427f493d25c65506a7"} Oct 30 19:50:12.322253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322225 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjx8t\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322267 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322339 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322384 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322521 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322579 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322654 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322602 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322682 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.322798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.322783 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.323961 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.323942 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:12.367226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.367182 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podStartSLOduration=2.36716906 podStartE2EDuration="2.36716906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:12.365544866 +0000 UTC m=+936.494778789" watchObservedRunningTime="2026-10-30 19:50:12.36716906 +0000 UTC m=+936.496403022" Oct 30 19:50:12.397447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.397382 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:12.409897 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.409875 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:12.423646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423622 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.423761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423662 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjx8t\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.423761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423693 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.423761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423732 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.423933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423783 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.423933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.423860 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424160 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424103 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424165 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424292 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424333 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424356 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424715 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424689 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.424918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.424902 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.426281 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.426253 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.426573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.426554 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.446374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.446343 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjx8t\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.447179 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.447157 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.451942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.451923 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:12.619955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.619919 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 19:50:12.625085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.625062 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637570 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637540 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637609 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637645 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637759 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637802 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.637854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637825 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.638086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637923 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.638086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.637968 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsbz\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.638086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.638009 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.659471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.659395 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 19:50:12.739247 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739207 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739289 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739321 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739424 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739453 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739510 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739543 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsbz\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.739657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739585 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.740221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.739741 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.740221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.740005 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.740221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.740030 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.740452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.740428 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.740548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.740476 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.742482 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.742461 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.742692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.742676 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.760878 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.760852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.761058 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.761042 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsbz\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:12.803380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.803351 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 19:50:12.803527 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:12.803472 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06298ea0_90dc_4a96_a90a_f9dde28fd618.slice/crio-dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab WatchSource:0}: Error finding container dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab: Status 404 returned error can't find the container with id dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab Oct 30 19:50:12.806701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.806655 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:12.806827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.806809 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:12.806919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.806896 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:12.939969 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:12.939555 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:13.270062 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.267917 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 19:50:13.286168 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.286102 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:13.286577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.286551 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:13.286787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.286764 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:13.317881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.317852 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" event={"ID":"06bcb85a-4bc2-4156-ab40-4b180f654402","Type":"ContainerStarted","Data":"7a4caafd4adea65938afd886c6ef9ae8c2c530eebcbb655665cd95d93f288d79"} Oct 30 19:50:13.319652 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.319611 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" event={"ID":"06298ea0-90dc-4a96-a90a-f9dde28fd618","Type":"ContainerStarted","Data":"a6890ac1f1119d8ddc9134dbfb8671b2d967825b59730f434655c986702520c3"} Oct 30 19:50:13.319802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.319672 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" event={"ID":"06298ea0-90dc-4a96-a90a-f9dde28fd618","Type":"ContainerStarted","Data":"dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab"} Oct 30 19:50:13.331718 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.331696 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:13.376551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.375521 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podStartSLOduration=1.375483171 podStartE2EDuration="1.375483171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:13.3746164 +0000 UTC m=+937.503850333" watchObservedRunningTime="2026-10-30 19:50:13.375483171 +0000 UTC m=+937.504717093" Oct 30 19:50:13.415379 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.415348 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.489629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.489388 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:13.490671 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.490649 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:13.490805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:13.490690 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:14.328195 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.328159 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" event={"ID":"06bcb85a-4bc2-4156-ab40-4b180f654402","Type":"ContainerStarted","Data":"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0"} Oct 30 19:50:14.460519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.458142 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:14.460519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.458205 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:14.940206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.940173 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:14.941851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.941822 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:14.941998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:14.941887 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:15.398667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:15.398635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.456519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:15.455968 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:15.456519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:15.456028 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:15.939864 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:15.939831 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:15.940071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:15.939879 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:16.106765 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:50:16.106728 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:50:16.461522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:16.460948 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:16.461522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:16.461012 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:16.940147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:16.940115 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:16.940326 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:16.940171 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:17.456646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:17.456607 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:17.456943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:17.456910 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:17.930466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:17.930430 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.940793 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:17.940769 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:17.940898 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:17.940815 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:18.307036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:18.306855 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:18.470910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:18.470875 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:50:18.471191 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:18.471163 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:50:18.939995 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:18.939954 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:18.940388 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:18.940019 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:19.707820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.707780 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:19.708744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.708718 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:19.709452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.709423 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 19:50:19.783478 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.783421 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podStartSLOduration=7.783406552 podStartE2EDuration="7.783406552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:14.403671957 +0000 UTC m=+938.532905892" watchObservedRunningTime="2026-10-30 19:50:19.783406552 +0000 UTC m=+943.912640472" Oct 30 19:50:19.940187 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.940149 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:50:19.940603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:19.940197 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:50:20.127710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:20.127632 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:20.944672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:20.944644 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:20.945141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:20.945119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:20.946030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:20.946010 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 19:50:21.324048 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:21.323960 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:21.340702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:21.340668 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:21.425641 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:21.425610 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:21.986295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:21.986265 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.697080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:23.697046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:25.879031 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:25.878993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.012249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.012218 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 19:50:27.016719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.016694 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.053289 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.053251 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 19:50:27.125008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.124949 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.125210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125187 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.125298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125249 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.125298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125279 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.125407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125322 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.126464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125411 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.126464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125525 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.126464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125608 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.126464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.125645 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqltc\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.161125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.161089 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.227143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227105 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227155 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqltc\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227222 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227244 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227278 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227305 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227342 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227426 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227532 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227656 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227697 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.227974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.227948 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.228391 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.228368 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.228697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.228673 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.229984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.229961 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.232211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.232187 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.252395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.252372 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.252715 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.252695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqltc\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.335561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.335290 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:27.713957 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:27.713924 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde39728e_03b2_43af_9df8_3106cb64fa61.slice/crio-f93f27209ceee32bf8ab5bfaa3b7915536adbf0a818cfa43ee3727e4566c139b WatchSource:0}: Error finding container f93f27209ceee32bf8ab5bfaa3b7915536adbf0a818cfa43ee3727e4566c139b: Status 404 returned error can't find the container with id f93f27209ceee32bf8ab5bfaa3b7915536adbf0a818cfa43ee3727e4566c139b Oct 30 19:50:27.714867 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.714844 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 19:50:27.716505 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.716462 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:27.716599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.716564 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:27.716638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:27.716604 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:28.021882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:28.021797 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:28.153006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:28.152977 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:28.472385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:28.472351 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" event={"ID":"de39728e-03b2-43af-9df8-3106cb64fa61","Type":"ContainerStarted","Data":"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13"} Oct 30 19:50:28.472578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:28.472414 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" event={"ID":"de39728e-03b2-43af-9df8-3106cb64fa61","Type":"ContainerStarted","Data":"f93f27209ceee32bf8ab5bfaa3b7915536adbf0a818cfa43ee3727e4566c139b"} Oct 30 19:50:28.567521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:28.566701 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podStartSLOduration=2.5666756790000003 podStartE2EDuration="2.566675679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.534038887 +0000 UTC m=+952.663272809" watchObservedRunningTime="2026-10-30 19:50:28.566675679 +0000 UTC m=+952.695909601" Oct 30 19:50:29.336127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:29.336097 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:29.337666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:29.337632 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:29.337778 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:29.337694 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:29.587866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:29.587790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.336473 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:30.336439 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.336911 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:30.336488 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:31.189900 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:31.189864 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.336955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:31.336919 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.337341 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:31.336981 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:32.336451 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:32.336420 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.336667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:32.336470 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:32.793047 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:32.793015 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.336966 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:33.336935 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.337138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:33.336994 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:34.336561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.336448 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.336955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.336551 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:50:34.373725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.373692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.862461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.862423 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 19:50:34.869749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.869715 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 19:50:34.869925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.869822 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.886915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.886868 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"kube-root-ca.crt\"" Oct 30 19:50:34.888424 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.888397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.899758 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.899736 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"openshift-service-ca.crt\"" Oct 30 19:50:34.899758 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.899736 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"default-dockercfg-ljjjh\"" Oct 30 19:50:34.899975 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.899955 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a3426\"/\"distinct-2-w4-8a3426-kserve-self-signed-certs\"" Oct 30 19:50:34.976263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976234 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.976432 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976289 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.976488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976456 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.976584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976565 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnm2f\" (UniqueName: \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.976637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976621 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:34.976712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:34.976697 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.080350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.080305 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.080695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.080672 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnm2f\" (UniqueName: \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.080838 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.080825 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.080951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.080939 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.081217 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.081203 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.083580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.083555 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.084230 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.083430 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.084403 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.082677 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.089987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.084035 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.104968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.104577 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.109540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.106779 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.109540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.107201 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnm2f\" (UniqueName: \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\") pod \"distinct-2-w4-8a3426-kserve-5784789f98-nb6w5\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.190285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.190249 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:35.340909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.340872 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:35.341363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.341237 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:35.342044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.342023 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 19:50:35.502466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.502441 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 19:50:35.505586 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:35.505557 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60e5ea21_98c1_4a84_9532_da59c98c4fe2.slice/crio-ccb52fe1c0d03cee504c59f8660b306cccdb544c337c59b60122e85dc88d4119 WatchSource:0}: Error finding container ccb52fe1c0d03cee504c59f8660b306cccdb544c337c59b60122e85dc88d4119: Status 404 returned error can't find the container with id ccb52fe1c0d03cee504c59f8660b306cccdb544c337c59b60122e85dc88d4119 Oct 30 19:50:35.526142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:35.526108 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" event={"ID":"60e5ea21-98c1-4a84-9532-da59c98c4fe2","Type":"ContainerStarted","Data":"ccb52fe1c0d03cee504c59f8660b306cccdb544c337c59b60122e85dc88d4119"} Oct 30 19:50:36.135211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:36.135180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:36.533279 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:36.533196 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" event={"ID":"60e5ea21-98c1-4a84-9532-da59c98c4fe2","Type":"ContainerStarted","Data":"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc"} Oct 30 19:50:36.610417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:36.610365 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podStartSLOduration=2.610351969 podStartE2EDuration="2.610351969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.609799718 +0000 UTC m=+960.739033640" watchObservedRunningTime="2026-10-30 19:50:36.610351969 +0000 UTC m=+960.739585889" Oct 30 19:50:37.191262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:37.191225 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:37.596618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:37.596538 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.091932 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:39.091902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.761367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:40.761333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.047302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.047267 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 19:50:42.052605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.052582 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.101307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.101272 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 19:50:42.188010 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.187971 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188116 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nxqn\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188197 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188153 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188316 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188279 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188375 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188336 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188375 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188366 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188438 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188563 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.188685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.188670 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.289819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.289775 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nxqn\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.289838 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.289956 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.289979 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290004 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290028 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290074 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290156 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290221 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290804 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290932 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290838 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290932 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290875 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.290932 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.290908 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.291284 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.291253 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.293008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.292987 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.293270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.293255 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.296694 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.296673 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 19:50:42.300898 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.300854 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.316357 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.316326 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nxqn\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.316827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.316806 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.335428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.334684 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 19:50:42.368768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.368732 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:42.391846 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.391812 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.391979 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.391865 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.391979 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.391892 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ddw\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392089 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392030 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392104 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392278 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392333 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392411 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392365 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.392461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.392421 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494374 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494407 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494434 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ddw\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494487 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494593 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494778 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494823 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494921 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494857 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.494979 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.494917 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.495757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.495192 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.495757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.495567 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.495757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.495662 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.495757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.495714 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.497744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.497722 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.498007 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.497982 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.521697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.521664 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.521826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.521744 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ddw\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\") pod \"e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.653137 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.653106 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:42.730869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.730371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 19:50:42.739295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.739258 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:42.739417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.739335 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:42.739417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.739365 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:42.919594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:42.919517 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:43.014851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.014822 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 19:50:43.025427 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:43.025397 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca576ba_0a1a_457c_be3a_c3bbaac08f0c.slice/crio-8fab0cdea99a1a302790e1a10a778eb3265f14a89f62c486c47d13bb0503e064 WatchSource:0}: Error finding container 8fab0cdea99a1a302790e1a10a778eb3265f14a89f62c486c47d13bb0503e064: Status 404 returned error can't find the container with id 8fab0cdea99a1a302790e1a10a778eb3265f14a89f62c486c47d13bb0503e064 Oct 30 19:50:43.029264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.029231 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:43.029426 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.029406 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:43.029533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.029520 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:43.588009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.587971 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" event={"ID":"75ee3675-99d4-4e43-bb3b-b63c04424db1","Type":"ContainerStarted","Data":"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635"} Oct 30 19:50:43.588482 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.588037 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" event={"ID":"75ee3675-99d4-4e43-bb3b-b63c04424db1","Type":"ContainerStarted","Data":"5550df82de81c4def46cc905956ede2ff6d8f5ff1d5502d2898d5e5702ffdae8"} Oct 30 19:50:43.589662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.589640 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" event={"ID":"cca576ba-0a1a-457c-be3a-c3bbaac08f0c","Type":"ContainerStarted","Data":"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc"} Oct 30 19:50:43.589735 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.589681 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" event={"ID":"cca576ba-0a1a-457c-be3a-c3bbaac08f0c","Type":"ContainerStarted","Data":"8fab0cdea99a1a302790e1a10a778eb3265f14a89f62c486c47d13bb0503e064"} Oct 30 19:50:43.667516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.663999 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podStartSLOduration=2.663978794 podStartE2EDuration="2.663978794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.660991371 +0000 UTC m=+967.790225304" watchObservedRunningTime="2026-10-30 19:50:43.663978794 +0000 UTC m=+967.793212716" Oct 30 19:50:43.667516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.666414 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:43.715014 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:43.714964 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podStartSLOduration=1.7149504599999998 podStartE2EDuration="1.71495046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:43.713686905 +0000 UTC m=+967.842920841" watchObservedRunningTime="2026-10-30 19:50:43.71495046 +0000 UTC m=+967.844184445" Oct 30 19:50:44.374735 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:44.374702 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:44.375179 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:44.375154 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:44.375383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:44.375357 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:44.686521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:44.681421 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:44.686521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:44.681522 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:45.197403 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.196673 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:45.211637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.211610 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:45.380518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.374536 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:45.380518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.374598 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:45.380518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.377486 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.631641 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.631566 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:45.662524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.656714 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:45.662764 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:45.662733 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:46.108626 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:50:46.108590 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:50:46.370575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:46.370031 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:46.370575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:46.370106 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:46.666519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:46.665981 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:46.666519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:46.666036 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:47.375521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:47.375235 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:47.375521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:47.375295 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:47.655001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:47.654968 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:47.655313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:47.655267 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:48.263521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.259522 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.277515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.275643 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.338519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.336783 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.390520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.387198 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:48.390520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.387255 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:48.631579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.627610 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.659521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.654084 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:48.659521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.654145 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:48.815520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.784602 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:48.930534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:48.927584 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:49.170520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.066419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:49.202577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.127075 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:50:49.253214 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.147902 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:50:49.270691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.270639 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:49.272380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.272360 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:50:49.303636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.283513 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:49.342724 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.340759 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:50:49.342724 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.341122 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:49.375945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.375915 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:49.380594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.380557 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:49.550978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.529790 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:50:49.724655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.713704 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:49.724993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.724937 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:49.729710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.729662 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:50:49.760531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:49.743151 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:50.181521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.174748 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.214567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.205458 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:50:50.258522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.246256 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.258522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.254166 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:50:50.315640 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.315591 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:50:50.316416 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.316382 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:50:50.386465 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.345581 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.386728 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.361285 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.413995 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.395896 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.501059 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.500165 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:50.547940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.547885 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627115 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627217 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627322 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627362 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627415 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:50:50.630522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.627451 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:50.665908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.665854 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 19:50:50.671770 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.671695 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 19:50:50.757286 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.757183 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.774342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.774298 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:50:50.786077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.786053 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:50:50.825315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.825286 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:50:50.903958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.902348 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:50:50.907597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.907572 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:50:50.908043 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.908027 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:50:50.948479 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.943575 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:50.956095 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.956030 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:50:50.997930 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:50.996514 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:50:51.007675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:51.007603 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:50:51.369899 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:51.369814 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:50:51.370152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:51.370125 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:50:52.148059 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.148023 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:52.149195 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.149175 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:52.150588 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.150569 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 19:50:52.373512 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.373467 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:52.373813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.373791 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:52.374510 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.374473 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 19:50:52.424032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.423956 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 19:50:52.428236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.428211 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.442973 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.442944 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 19:50:52.459247 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.459208 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488098 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488162 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n442h\" (UniqueName: \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488292 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488337 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488433 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.489955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.488475 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.509617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.509576 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f349a1\"/\"distinct-2-w6-f349a1-kserve-self-signed-certs\"" Oct 30 19:50:52.589237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589200 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589247 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n442h\" (UniqueName: \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589308 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589436 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589576 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589610 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589608 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589802 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.589918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.589898 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.591794 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.591758 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.592229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.592211 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.628513 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.628472 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n442h\" (UniqueName: \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\") pod \"distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:52.748050 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:52.747955 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:53.175820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:53.175797 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 19:50:53.176334 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:53.176312 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaaecb09e_0a11_4b21_b333_2f853774f3b5.slice/crio-6f7b309672a5c17c81532a4538d0ca52a818db39ca034b5da5f8a1fb781dccae WatchSource:0}: Error finding container 6f7b309672a5c17c81532a4538d0ca52a818db39ca034b5da5f8a1fb781dccae: Status 404 returned error can't find the container with id 6f7b309672a5c17c81532a4538d0ca52a818db39ca034b5da5f8a1fb781dccae Oct 30 19:50:54.173450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:54.173413 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" event={"ID":"aaecb09e-0a11-4b21-b333-2f853774f3b5","Type":"ContainerStarted","Data":"ed86fb8ddecc6706f5ab99bebd27a2f344b95eb3935fcc156c13dee21b7fb59e"} Oct 30 19:50:54.173649 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:54.173465 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" event={"ID":"aaecb09e-0a11-4b21-b333-2f853774f3b5","Type":"ContainerStarted","Data":"6f7b309672a5c17c81532a4538d0ca52a818db39ca034b5da5f8a1fb781dccae"} Oct 30 19:50:54.223812 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:54.223757 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podStartSLOduration=2.223743334 podStartE2EDuration="2.223743334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:54.222838584 +0000 UTC m=+978.352072512" watchObservedRunningTime="2026-10-30 19:50:54.223743334 +0000 UTC m=+978.352977255" Oct 30 19:50:54.413008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:54.411188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:54.748353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:54.748324 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:50:56.537282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:56.537240 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:57.511526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.511459 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 19:50:57.515978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.515955 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.539531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.539475 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 19:50:57.556048 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556017 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556095 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556156 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhm7\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556296 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556462 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556443 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556536 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556623 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556602 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556682 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556652 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.556742 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.556699 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658443 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658402 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658465 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658568 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658591 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhm7\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658638 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658779 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658820 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.658908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658882 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658936 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658942 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.658942 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.659055 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.659195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.659617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.659594 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.661222 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.661199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.661353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.661272 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.680513 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.680471 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhm7\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.680645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.680477 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\") pod \"e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:57.832973 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:57.832892 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:58.075718 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.075688 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:50:58.081836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.081805 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.120618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.120561 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:50:58.171763 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.171724 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.171957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.171903 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172026 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172004 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172215 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172263 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172296 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172346 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172383 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.172477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.172434 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.174867 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.174844 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 19:50:58.175442 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:58.175419 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2236e32f_fdda_42e5_84e7_a0b14a6f8f87.slice/crio-47423c9986074a1be3c7ae95f3d0abedda43fd01011f9665af8c12d000bb2102 WatchSource:0}: Error finding container 47423c9986074a1be3c7ae95f3d0abedda43fd01011f9665af8c12d000bb2102: Status 404 returned error can't find the container with id 47423c9986074a1be3c7ae95f3d0abedda43fd01011f9665af8c12d000bb2102 Oct 30 19:50:58.177925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.177890 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:58.178038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.177958 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:58.178038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.177988 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:58.209991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.209960 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" event={"ID":"2236e32f-fdda-42e5-84e7-a0b14a6f8f87","Type":"ContainerStarted","Data":"47423c9986074a1be3c7ae95f3d0abedda43fd01011f9665af8c12d000bb2102"} Oct 30 19:50:58.273728 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273695 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.273910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273778 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.273910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.273910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273883 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273934 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.273967 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274002 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274063 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274134 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274621 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274756 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.274875 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274849 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.275046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274971 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.275046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.274971 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.276456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.276435 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.276566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.276549 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.296196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.296155 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.296338 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.296314 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.397153 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.397121 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:58.601452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.601414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:58.737897 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:50:58.737869 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b71e00_bfeb_48ff_ac4d_7178b78442cf.slice/crio-13c0895e3cfd273394f433785320b77cf7132dcdab5ab4a1c0f15a3db1f460e6 WatchSource:0}: Error finding container 13c0895e3cfd273394f433785320b77cf7132dcdab5ab4a1c0f15a3db1f460e6: Status 404 returned error can't find the container with id 13c0895e3cfd273394f433785320b77cf7132dcdab5ab4a1c0f15a3db1f460e6 Oct 30 19:50:58.738280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.738258 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:50:58.740787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.740752 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:58.740889 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.740834 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:58.740889 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:58.740879 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:50:59.218464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.218426 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" event={"ID":"64b71e00-bfeb-48ff-ac4d-7178b78442cf","Type":"ContainerStarted","Data":"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8"} Oct 30 19:50:59.218673 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.218541 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" event={"ID":"64b71e00-bfeb-48ff-ac4d-7178b78442cf","Type":"ContainerStarted","Data":"13c0895e3cfd273394f433785320b77cf7132dcdab5ab4a1c0f15a3db1f460e6"} Oct 30 19:50:59.220360 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.220330 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" event={"ID":"2236e32f-fdda-42e5-84e7-a0b14a6f8f87","Type":"ContainerStarted","Data":"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3"} Oct 30 19:50:59.282594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.282535 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podStartSLOduration=1.282519862 podStartE2EDuration="1.282519862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:59.280800145 +0000 UTC m=+983.410034104" watchObservedRunningTime="2026-10-30 19:50:59.282519862 +0000 UTC m=+983.411753780" Oct 30 19:50:59.329394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.329340 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podStartSLOduration=2.32932631 podStartE2EDuration="2.32932631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:59.326898916 +0000 UTC m=+983.456132847" watchObservedRunningTime="2026-10-30 19:50:59.32932631 +0000 UTC m=+983.458560230" Oct 30 19:50:59.397357 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.397324 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:50:59.398767 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.398739 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:50:59.398903 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.398799 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:50:59.834018 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.833987 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:50:59.836659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.836633 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:50:59.836797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:50:59.836691 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:00.398988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:00.398953 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:00.399294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:00.399262 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:00.431428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:00.431388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:00.833548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:00.833460 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:00.833548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:00.833522 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:01.399410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:01.397782 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:01.399410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:01.397857 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:01.833616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:01.833539 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:01.833616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:01.833584 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:02.397624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.397574 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:02.397858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.397646 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:02.446644 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.446610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:02.760580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.757798 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:51:02.808143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.808107 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:51:02.834135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.834102 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:02.834322 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:02.834163 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:03.309642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:03.309615 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:51:03.397813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:03.397767 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:03.398116 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:03.398071 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:03.833162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:03.833128 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:03.833567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:03.833177 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:04.391862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:04.391824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:04.397398 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:04.397369 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:04.397561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:04.397422 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:04.834061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:04.833974 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:51:04.834061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:04.834024 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:51:05.397665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:05.397629 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:51:05.397861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:05.397692 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 19:51:05.838189 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:05.838115 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:51:05.838632 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:05.838614 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:51:05.839315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:05.839296 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 19:51:06.012623 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:06.012578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.512807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:06.512775 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:51:06.513611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:06.513586 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:51:06.514936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:06.514915 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:51:07.665001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:07.664968 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:09.188348 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:09.188312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.009149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:10.009078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.922523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:10.922477 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.732001 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:12.731972 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.879770 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:12.879731 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 19:51:12.885619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:12.885594 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:12.927035 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:12.926999 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 19:51:13.030776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.030684 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.030776 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.030744 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031024 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.030838 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031034 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hcl\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031081 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031194 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031158 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031245 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031218 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031286 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031275 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.031327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.031308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132525 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132664 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hcl\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132691 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132712 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132786 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132816 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132862 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.132974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132933 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132976 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.132995 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.133026 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.133166 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.133197 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.133682 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.133660 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.135532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.135478 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.135749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.135720 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.147536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.147515 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 19:51:13.152891 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.152867 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.173833 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.173805 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.175586 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.175566 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hcl\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\") pod \"e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.183534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.183488 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 19:51:13.229787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.229758 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:13.335395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335361 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335433 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335453 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335470 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwgb\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335530 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335674 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335699 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.335869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335742 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.336029 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.335865 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437575 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437609 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437633 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwgb\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437762 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437928 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437889 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.437928 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437917 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.437977 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438021 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438055 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438116 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438273 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438242 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438341 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438287 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438384 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.438775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.438750 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.440075 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.440054 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.440523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.440473 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.458366 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.458339 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.458598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.458580 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwgb\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\") pod \"e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.483687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.483663 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:13.551653 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:51:13.551616 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eb72b2_1541_4c2c_8e15_48a27a19752b.slice/crio-beee01306c3a8e28c7886859b9cfe6e152696745ccdccfe968c28fb41658b700 WatchSource:0}: Error finding container beee01306c3a8e28c7886859b9cfe6e152696745ccdccfe968c28fb41658b700: Status 404 returned error can't find the container with id beee01306c3a8e28c7886859b9cfe6e152696745ccdccfe968c28fb41658b700 Oct 30 19:51:13.552132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.551876 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 19:51:13.554485 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.554444 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:13.554636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.554549 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:13.554636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.554601 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:13.790327 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:51:13.790301 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a80b343_468d_46fe_b798_9bb1bcf76fd1.slice/crio-d8bbb2dec5f95d78dce29819cb01d8bdeea7b450a205bc9fa0ca0ab4840df0d8 WatchSource:0}: Error finding container d8bbb2dec5f95d78dce29819cb01d8bdeea7b450a205bc9fa0ca0ab4840df0d8: Status 404 returned error can't find the container with id d8bbb2dec5f95d78dce29819cb01d8bdeea7b450a205bc9fa0ca0ab4840df0d8 Oct 30 19:51:13.792380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.792355 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 19:51:13.793120 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.793092 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:13.793188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.793154 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:13.793225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:13.793185 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:14.033666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.033601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:14.145935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.145898 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:14.384757 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.384724 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" event={"ID":"8a80b343-468d-46fe-b798-9bb1bcf76fd1","Type":"ContainerStarted","Data":"59e75d016e09722c64b6f450a9deb67695ab2e8cc7cd153c8ed7c6eb676b254c"} Oct 30 19:51:14.384973 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.384789 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" event={"ID":"8a80b343-468d-46fe-b798-9bb1bcf76fd1","Type":"ContainerStarted","Data":"d8bbb2dec5f95d78dce29819cb01d8bdeea7b450a205bc9fa0ca0ab4840df0d8"} Oct 30 19:51:14.386463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.386437 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" event={"ID":"90eb72b2-1541-4c2c-8e15-48a27a19752b","Type":"ContainerStarted","Data":"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72"} Oct 30 19:51:14.386591 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.386477 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" event={"ID":"90eb72b2-1541-4c2c-8e15-48a27a19752b","Type":"ContainerStarted","Data":"beee01306c3a8e28c7886859b9cfe6e152696745ccdccfe968c28fb41658b700"} Oct 30 19:51:14.453479 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.448979 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podStartSLOduration=1.44896053 podStartE2EDuration="1.44896053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.446473943 +0000 UTC m=+998.575707865" watchObservedRunningTime="2026-10-30 19:51:14.44896053 +0000 UTC m=+998.578194454" Oct 30 19:51:14.484449 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.484420 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:14.486051 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.486030 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:14.486140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.486079 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:14.505243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.505189 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podStartSLOduration=2.505175673 podStartE2EDuration="2.505175673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:14.503092838 +0000 UTC m=+998.632326761" watchObservedRunningTime="2026-10-30 19:51:14.505175673 +0000 UTC m=+998.634409593" Oct 30 19:51:14.613012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:14.612984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.230877 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:15.230844 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:15.232309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:15.232290 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.232406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:15.232332 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:15.487577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:15.484894 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.487577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:15.484955 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:16.111646 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:51:16.111607 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:51:16.230553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:16.230473 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.230738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:16.230584 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:16.487520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:16.486976 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.487520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:16.487039 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:16.983521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:16.970931 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:17.230285 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:17.230253 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.230438 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:17.230304 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:17.484902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:17.484862 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.485213 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:17.485180 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:18.230880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:18.230845 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.231380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:18.230910 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:18.484994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:18.484897 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.485272 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:18.485243 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:19.230904 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:19.230874 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.231394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:19.230929 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:19.486581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:19.485936 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.486581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:19.485999 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:19.847283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:19.847188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.230985 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:20.230943 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.231417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:20.231014 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 19:51:20.384557 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:20.384057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.484429 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:20.484330 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.484429 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:20.484398 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:21.484702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:21.484658 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 19:51:21.485235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:21.484743 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 19:51:21.809949 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:21.804146 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:21.809949 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:21.805663 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:21.812067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:21.811450 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 19:51:22.752837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:22.752801 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:22.754054 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:22.754024 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:22.758055 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:22.757206 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 19:51:23.031785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:23.031694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:24.977300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:24.977269 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:26.686731 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:26.686693 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.277597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:28.277559 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.395488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:28.395450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.504955 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:28.504897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:30.097515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:30.097454 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.956392 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:31.956361 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:33.899172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:33.899136 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.489842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:35.489815 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.103423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:37.103392 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:38.697712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:38.697684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.658361 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:40.658328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.265127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.265092 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:51:42.271240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.271216 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.297061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.297032 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:51:42.299752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.299718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.352013 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.351984 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352132 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352109 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352187 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352178 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352205 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352292 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352233 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352383 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352369 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352426 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352392 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352426 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352418 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzw8\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.352617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.352596 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.460143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.460102 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.460431 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.460415 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.460693 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.460671 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.460970 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.460919 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.461154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.461136 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.461295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.461278 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.461475 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.461436 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.471621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.469758 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.472316 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.470345 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.473460 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.470600 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.487353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.470827 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.487677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.471056 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.487836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.473358 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.488027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.487990 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzw8\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.488895 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.482388 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.489314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.489274 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.510243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.510220 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.510427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.510399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzw8\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\") pod \"e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.586737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.586670 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:42.920308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.920283 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:51:42.921099 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.921069 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:42.921163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.921129 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:42.921163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:42.921158 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:51:43.670517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:43.667343 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" event={"ID":"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3","Type":"ContainerStarted","Data":"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6"} Oct 30 19:51:43.670517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:43.667422 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" event={"ID":"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3","Type":"ContainerStarted","Data":"fdab9973bf4213ba58ce64c0743a97880a969b141f149ff22ae8fe9c5615ff0c"} Oct 30 19:51:43.730315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:43.730248 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podStartSLOduration=1.730233651 podStartE2EDuration="1.730233651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:43.726712528 +0000 UTC m=+1027.855946449" watchObservedRunningTime="2026-10-30 19:51:43.730233651 +0000 UTC m=+1027.859467571" Oct 30 19:51:44.212949 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:44.212895 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.587602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:44.587484 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:44.588897 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:44.588863 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:44.589014 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:44.588921 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:45.587359 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:45.587327 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:45.587801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:45.587378 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:46.085543 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:51:46.085512 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:51:46.242530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:46.242480 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:46.587568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:46.587539 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:46.587977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:46.587591 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:47.587256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:47.587225 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:47.587436 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:47.587275 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:48.006606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:48.006573 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.587155 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:48.587120 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:48.587340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:48.587184 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:49.587511 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:49.587451 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:51:49.587918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:49.587533 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 19:51:49.835560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:49.824338 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.402854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.402806 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.604646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.604602 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:51.604854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.604671 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.832696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.831056 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.832696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.832298 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.916653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.916607 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.964654 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.964606 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:51.985834 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:51.985792 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.018523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.015997 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.018523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.017162 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.267518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.266645 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.330582 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.329600 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.464732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.464682 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.493476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.491332 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:51:52.704298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.704251 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:52.705110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.704667 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:52.868521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.861171 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:51:52.868521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.861557 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:51:52.868521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.862078 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:51:52.868521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.863020 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:51:52.909620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.909576 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.960455 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.960536 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.981691 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:52.982226 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.005648 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:51:53.058639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.005727 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:51:53.087791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.062383 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.087791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.080129 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:51:53.089875 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.089843 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.155712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.129549 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:51:53.155712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.090821 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:51:53.155712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.146612 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:51:53.286661 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.286605 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:53.302530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.287450 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:51:53.523609 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.462444 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 19:51:54.061063 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.529901 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:51:54.084523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.577413 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:54.084844 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.084804 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.085016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.810176 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:51:54.104650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.861611 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.108523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.882618 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:51:54.112711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.882668 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.120857 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.882849 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.121126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.953256 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.123628 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.982221 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.124005 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.994852 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:51:54.124287 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:53.622379 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 19:51:54.153024 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.152975 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.158533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.124879 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.158533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.157079 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:51:54.158533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.157369 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:54.158533 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.157414 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.296596 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.287049 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:54.725663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:54.591105 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:51:55.079580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.075242 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.153522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.153306 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:51:55.171138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.153827 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.171138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.155268 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.171138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.157565 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.171138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.159265 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.199652 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.199612 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.207650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.207588 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:51:55.270916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.234353 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.270916 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.234390 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:51:55.297041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.296997 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.325536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.325091 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:55.325536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.325156 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.352284 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.339481 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:51:55.434553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.413588 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.467318 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.467183 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:55.467318 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.467245 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.687685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.687577 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:55.754184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.754149 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:51:55.754714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.754526 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:51:55.777990 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.777054 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:51:55.804517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:55.804467 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:51:56.098587 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.094910 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:51:56.171518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.158222 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.171518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.160656 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:51:56.205722 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.202582 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.297520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.289600 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.667176 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.290998 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.678732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.343659 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.685523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.510286 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:56.685523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.680476 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:56.685523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:56.680545 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.095363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.095270 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:51:57.179880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.163771 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:51:57.205536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.205489 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:57.205920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.205808 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:51:57.206865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.194699 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.224455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.214163 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.224455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.215707 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:51:57.224455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.215957 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:57.244306 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.244276 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:51:57.275651 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.275612 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:51:57.304695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.304663 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:51:57.305112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.305090 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:51:57.305229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.305191 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:51:57.359769 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:51:57.359687 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:52:00.134329 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:00.134300 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.367216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:02.367178 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.999083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:03.999056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.512669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:05.512621 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:07.271153 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:07.271121 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:08.811138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:08.811103 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.510092 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:10.510049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.081014 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.080975 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.283515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.283353 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 19:52:12.289516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.288717 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.328612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.321330 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 19:52:12.398853 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.398819 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399055 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399000 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399130 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399104 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhx5r\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399186 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399399 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399368 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399434 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399560 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399600 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.399785 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.399654 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501421 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhx5r\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501473 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501617 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501686 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501719 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501746 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501792 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501840 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501857 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.501923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.501892 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.502088 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.502071 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.502264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.502241 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.502433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.502416 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.504148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.504124 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.504524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.504478 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.524977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.524948 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhx5r\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.525090 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.524951 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\") pod \"e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.615592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.615559 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:12.777826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.777797 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 19:52:12.782224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.782203 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.828575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.828545 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 19:52:12.874084 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.874052 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 19:52:12.878667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.878644 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:12.908310 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908278 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908393 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908461 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908479 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908612 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908666 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljdk\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908730 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908715 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908762 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.908819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.908808 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:12.979708 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:12.979532 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 19:52:13.010731 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010695 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.010731 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010735 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.010968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010769 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.010968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010805 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.010968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010829 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.010968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.010863 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.011110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011085 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 19:52:13.011215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011196 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.011256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011221 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.011370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011304 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljdk\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.011432 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011386 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.011643 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011618 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.011765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011715 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.011892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.011872 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.012082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012063 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.012128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012119 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.012183 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012172 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.012252 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012241 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.012306 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012296 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.012386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012321 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.012386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012349 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcx7\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.012633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012448 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.012633 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012537 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.012957 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.012938 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.014087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.014068 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.015971 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.015916 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.020973 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:13.020951 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be64621_aa39_4baa_aad0_c669440ee371.slice/crio-6a824feb4f9e8424f4fd78a1d1cdbf5106d3382dcbafab02cdc7ad53804f977d WatchSource:0}: Error finding container 6a824feb4f9e8424f4fd78a1d1cdbf5106d3382dcbafab02cdc7ad53804f977d: Status 404 returned error can't find the container with id 6a824feb4f9e8424f4fd78a1d1cdbf5106d3382dcbafab02cdc7ad53804f977d Oct 30 19:52:13.023630 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.023596 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.023711 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.023664 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.023759 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.023710 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.039250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.039202 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" event={"ID":"8be64621-aa39-4baa-aad0-c669440ee371","Type":"ContainerStarted","Data":"6a824feb4f9e8424f4fd78a1d1cdbf5106d3382dcbafab02cdc7ad53804f977d"} Oct 30 19:52:13.068370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.068012 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljdk\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.068370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.068014 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.098447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.098416 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:13.113807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.113772 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.113939 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.113840 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.113939 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.113881 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.113939 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.113918 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.114134 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.114090 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.114227 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.114210 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.114295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.114279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.114358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.114342 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcx7\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.114437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.114422 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.115271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.115041 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.115271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.115052 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.115865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.115835 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.116399 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.116377 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.117576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.117510 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.119698 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.119584 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.125413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.125393 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.141777 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.141742 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.143982 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.143956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcx7\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\") pod \"e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.198019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.197745 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:13.446066 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:13.446039 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d22d3b6_dfaa_4cfb_969f_e3950f569730.slice/crio-22fc3cd86096d6d7c66211035ecd9f25aa7543912b46534f52dace1e951fa549 WatchSource:0}: Error finding container 22fc3cd86096d6d7c66211035ecd9f25aa7543912b46534f52dace1e951fa549: Status 404 returned error can't find the container with id 22fc3cd86096d6d7c66211035ecd9f25aa7543912b46534f52dace1e951fa549 Oct 30 19:52:13.448696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.448662 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.448789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.448730 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.448789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.448760 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.449115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.449084 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 19:52:13.543216 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:13.543185 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff97fb2a_8949_4354_a737_0b40e026b6b1.slice/crio-7f88f9a4a5c98b5531ff4cc21c29308bb949e8f4205e68e3061c8375c7e2f8a8 WatchSource:0}: Error finding container 7f88f9a4a5c98b5531ff4cc21c29308bb949e8f4205e68e3061c8375c7e2f8a8: Status 404 returned error can't find the container with id 7f88f9a4a5c98b5531ff4cc21c29308bb949e8f4205e68e3061c8375c7e2f8a8 Oct 30 19:52:13.544890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.544864 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 19:52:13.546720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.546681 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.546847 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.546763 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:13.546847 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:13.546802 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:14.055521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.053684 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" event={"ID":"8be64621-aa39-4baa-aad0-c669440ee371","Type":"ContainerStarted","Data":"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f"} Oct 30 19:52:14.071524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.067999 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" event={"ID":"ff97fb2a-8949-4354-a737-0b40e026b6b1","Type":"ContainerStarted","Data":"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911"} Oct 30 19:52:14.071524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.068064 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" event={"ID":"ff97fb2a-8949-4354-a737-0b40e026b6b1","Type":"ContainerStarted","Data":"7f88f9a4a5c98b5531ff4cc21c29308bb949e8f4205e68e3061c8375c7e2f8a8"} Oct 30 19:52:14.095513 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.088393 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" event={"ID":"0d22d3b6-dfaa-4cfb-969f-e3950f569730","Type":"ContainerStarted","Data":"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768"} Oct 30 19:52:14.095513 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.088449 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" event={"ID":"0d22d3b6-dfaa-4cfb-969f-e3950f569730","Type":"ContainerStarted","Data":"22fc3cd86096d6d7c66211035ecd9f25aa7543912b46534f52dace1e951fa549"} Oct 30 19:52:14.103101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.101759 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:14.103101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.101920 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.103101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.101968 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:14.135011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.134964 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podStartSLOduration=2.134950675 podStartE2EDuration="2.134950675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:14.134351762 +0000 UTC m=+1058.263585684" watchObservedRunningTime="2026-10-30 19:52:14.134950675 +0000 UTC m=+1058.264184595" Oct 30 19:52:14.203239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.203206 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.213524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.207060 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:14.215165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.209758 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.215398 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.215371 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:14.275897 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.275839 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podStartSLOduration=2.275804443 podStartE2EDuration="2.275804443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:14.275206488 +0000 UTC m=+1058.404440412" watchObservedRunningTime="2026-10-30 19:52:14.275804443 +0000 UTC m=+1058.405038365" Oct 30 19:52:14.278742 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.278688 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podStartSLOduration=2.278657345 podStartE2EDuration="2.278657345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:14.206636815 +0000 UTC m=+1058.335870733" watchObservedRunningTime="2026-10-30 19:52:14.278657345 +0000 UTC m=+1058.407891267" Oct 30 19:52:14.626472 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.626431 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:14.626942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.626917 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:14.627174 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:14.627134 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:15.109523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.103973 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.109523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.104031 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:15.211017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.210979 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.211306 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.211274 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:15.624746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.624710 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:15.625153 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:15.625119 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:16.124660 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.124626 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:16.125220 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.125191 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:16.138871 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:52:16.138836 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:52:16.210522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.207625 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:16.210522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.207692 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:16.490997 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.490929 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.618109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.618076 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:16.618417 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:16.618386 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:17.105541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.100389 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.105541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.100469 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:17.207536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.198836 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.207536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.198900 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:17.632519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.632079 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.632519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:17.632141 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:18.187522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.171433 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.187522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.171628 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:18.267093 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.257471 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.267093 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.257585 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:18.639754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.638811 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.639754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:18.638879 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:19.129404 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.129311 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.129751 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.129716 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:19.208524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.204638 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.208841 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.208808 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:19.617862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.617829 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.618401 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.618370 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:52:19.712936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:19.712710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.099097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:20.099065 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.099251 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:20.099116 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 19:52:20.199544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:20.199477 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.199700 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:20.199568 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:52:21.153521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.152716 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:21.168423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.155131 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:21.168423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.164796 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 19:52:21.632865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.632825 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:21.633857 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.633822 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:21.635079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.635059 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 19:52:21.742870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.742838 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:21.744128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.744104 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:21.762748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.762718 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 19:52:21.780475 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:21.780437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:23.769717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:23.769679 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.465602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:25.465571 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:27.202663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:27.202624 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:28.765521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:28.757822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:29.113574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.113453 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.130764 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.130635 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.167373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.165586 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.167373 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.165979 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.240612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.240560 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.290540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.287444 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:29.694516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.693766 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:29.758525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.758014 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:29.787518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:29.786821 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:52:30.084608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.083949 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.114612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.114569 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:30.116163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.116139 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:52:30.140256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.137680 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:52:30.140256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.138869 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:52:30.159860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.159819 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.177520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.173597 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.177520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.174012 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:30.177520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.174337 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:52:30.177520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.174392 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:52:30.408583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.354864 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:30.408922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.371415 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.522521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.413343 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:52:30.522521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.413717 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:52:30.778747 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.778637 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.804179 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.804133 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:30.810243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.810208 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:52:30.827521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.815967 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:52:30.988519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:30.983029 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.008559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.006697 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:52:31.042582 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.011889 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.170779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.170727 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.201670 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.201625 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:52:31.214282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.214251 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:52:31.384586 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.369065 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:31.507948 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.507908 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:52:31.589819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.536007 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.674797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.650281 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:31.725083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.725041 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:31.754428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.754386 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.782559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.771666 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:52:31.869523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.868871 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:31.869523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.869251 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:52:32.002584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:31.991322 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.240521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.172786 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.250593 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.242193 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.438566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.393856 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.610528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.536704 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:52:32.610528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.540092 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:52:32.662523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.653459 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:52:32.662523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.653845 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 19:52:32.796597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.788294 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:52:32.796597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.788635 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:52:32.946536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.927595 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:32.946536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:32.928399 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:52:33.008518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.007645 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.244663 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.244618 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:52:33.290521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.279059 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.417545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.417469 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.439872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.439829 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:52:33.441206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.441179 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:52:33.556263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.556144 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.557951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.556895 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.723531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.699080 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.865585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.814067 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 19:52:33.865585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.814364 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:52:33.865585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.814747 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.965325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.939469 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:33.965325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.940168 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:52:33.965325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:33.940486 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:52:34.011986 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.011940 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.277536 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.270627 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:52:34.447152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.447106 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.447836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.447423 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.711675 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.711625 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:52:34.712262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.712236 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:52:34.740520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.725085 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:52:34.786522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.745216 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.834125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.834074 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.834783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.834750 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:52:34.942184 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.942143 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.943758 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:34.942450 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:52:35.097578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.096821 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:35.239385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.239350 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:52:35.239709 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.239667 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.272890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.272853 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:52:35.414572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.413792 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 30 19:52:35.452881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.452822 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:52:35.494238 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.494197 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.607616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.607575 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 19:52:35.700581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.700442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:35.737537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.712541 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.737537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.712488 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.795942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.769306 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:35.843818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.843757 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:52:35.848786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.848758 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:35.918382 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.918352 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:52:35.947159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:35.947107 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:36.049686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.049406 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:36.098582 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.098543 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:52:36.146458 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.146386 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:52:36.178310 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.178279 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:52:36.202588 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.200375 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:52:36.235861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.234665 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:52:36.237199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.236093 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:52:36.272455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.272402 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:52:36.342139 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.340434 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:52:36.365552 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:36.363643 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:52:38.341878 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:38.341840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.988725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:38.988689 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:52:38.989386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:38.988935 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" containerID="cri-o://7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d" gracePeriod=30 Oct 30 19:52:41.304909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:41.304810 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:42.882151 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:42.882115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:44.275166 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.275095 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:52:44.301883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.301853 4959 generic.go:356] "Generic (PLEG): container finished" podID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerID="7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d" exitCode=0 Oct 30 19:52:44.302011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.301890 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" event={"ID":"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4","Type":"ContainerDied","Data":"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d"} Oct 30 19:52:44.302011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.301909 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" Oct 30 19:52:44.302011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.301952 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4" event={"ID":"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4","Type":"ContainerDied","Data":"d3619d2066009fe57718301e23855d03a052aa023403956f42f5b6c31cc91167"} Oct 30 19:52:44.302011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.301983 4959 scope.go:122] "RemoveContainer" containerID="7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d" Oct 30 19:52:44.311784 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.311767 4959 scope.go:122] "RemoveContainer" containerID="7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d" Oct 30 19:52:44.312036 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:52:44.312019 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d\": container with ID starting with 7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d not found: ID does not exist" containerID="7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d" Oct 30 19:52:44.312095 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.312045 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d"} err="failed to get container status \"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d\": rpc error: code = NotFound desc = could not find container \"7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d\": container with ID starting with 7b441429d884e8bf65a4c89bcb0882a224fd51d1fd7ec365011d4c0fb1126c4d not found: ID does not exist" Oct 30 19:52:44.361662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361630 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.361831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361732 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.361831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361771 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.361831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361801 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361852 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.361982 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.362097 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362201 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.362184 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.362288 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\") pod \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\" (UID: \"a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4\") " Oct 30 19:52:44.362828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.362797 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:44.363008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.362978 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:44.363163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.363104 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.363240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.363213 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:44.363471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.363448 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:44.363590 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.363467 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:44.367645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.367079 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "kube-api-access-t4bw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:44.367645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.367148 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:44.367645 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.367179 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:44.371742 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.371678 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token" pod "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" (UID: "a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:44.464788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464752 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4bw9\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-kube-api-access-t4bw9\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464790 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464805 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464818 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464832 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464845 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464858 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.464981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.464870 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:52:44.583027 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.582929 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:44.695106 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.688453 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:52:44.697526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:44.696007 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-stmx4"] Oct 30 19:52:45.996949 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:45.996915 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" path="/var/lib/kubelet/pods/a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4/volumes" Oct 30 19:52:46.065847 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:52:46.065814 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:52:46.104573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:46.104536 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.259677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:46.246561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.449740 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:46.449706 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.574925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:46.574850 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.950520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:46.950466 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.068872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:47.068842 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.179416 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:47.179382 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.891857 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:47.891826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:49.668872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:49.668824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:50.140690 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:50.140658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:50.252687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:50.252662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:51.615161 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:51.615134 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.459416 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.459377 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.578037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.578002 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:52:52.579069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.579045 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" Oct 30 19:52:52.579069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.579069 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" Oct 30 19:52:52.579262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.579244 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c4fcbd-d5f6-405d-9f04-fa1c2f3964e4" containerName="istio-proxy" Oct 30 19:52:52.583802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.583779 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:52.598854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.598825 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:52:52.607265 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.607230 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ffc83e66\"" Oct 30 19:52:52.607504 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.607475 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ffc83e66-dockercfg-bwvwr\"" Oct 30 19:52:52.635408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.635377 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:52:52.635903 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.635692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.639946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.639922 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.662974 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.662930 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:52:52.886626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.856685 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:52.886626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.858655 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqbn\" (UniqueName: \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.886626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.858746 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.886626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.858771 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhz5h\" (UniqueName: \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:52.967338 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.967271 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqbn\" (UniqueName: \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.967674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.967655 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.967820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.967805 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhz5h\" (UniqueName: \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:52.968127 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.968112 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:52.968710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.968660 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:52.968993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:52.968969 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:53.040813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.040778 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhz5h\" (UniqueName: \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\") pod \"payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:53.045207 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.045178 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqbn\" (UniqueName: \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\") pod \"payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:53.202428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.202341 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:53.260385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.259921 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:53.408703 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.408665 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:53.605368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.604668 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:52:53.607729 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:53.607692 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25c63a8_59e3_4d43_9f92_c5d0c32dc1e3.slice/crio-09c71c00342a862847e3479b1066b82114cbde34e8d848834b78fbb1b81f86f6 WatchSource:0}: Error finding container 09c71c00342a862847e3479b1066b82114cbde34e8d848834b78fbb1b81f86f6: Status 404 returned error can't find the container with id 09c71c00342a862847e3479b1066b82114cbde34e8d848834b78fbb1b81f86f6 Oct 30 19:52:53.647394 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:53.647354 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981f87e9_f3a5_4f9c_8a54_59549a463c62.slice/crio-f00038ba1138873c0a36b9128f5e11e1716adba86efaee8e0dfd58b4b8ad6ca4 WatchSource:0}: Error finding container f00038ba1138873c0a36b9128f5e11e1716adba86efaee8e0dfd58b4b8ad6ca4: Status 404 returned error can't find the container with id f00038ba1138873c0a36b9128f5e11e1716adba86efaee8e0dfd58b4b8ad6ca4 Oct 30 19:52:53.648178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:53.648153 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:52:54.054940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.054909 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.173803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.173765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.371735 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.371639 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" event={"ID":"981f87e9-f3a5-4f9c-8a54-59549a463c62","Type":"ContainerStarted","Data":"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395"} Oct 30 19:52:54.371905 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.371745 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:52:54.371905 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.371771 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" event={"ID":"981f87e9-f3a5-4f9c-8a54-59549a463c62","Type":"ContainerStarted","Data":"f00038ba1138873c0a36b9128f5e11e1716adba86efaee8e0dfd58b4b8ad6ca4"} Oct 30 19:52:54.373637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.373609 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" event={"ID":"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3","Type":"ContainerStarted","Data":"6e091f93036bd9302ffe9a64a138b8bbabd6f0d2270dd06cd6c219f7777e1ff9"} Oct 30 19:52:54.373789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.373668 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:52:54.373789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.373693 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" event={"ID":"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3","Type":"ContainerStarted","Data":"09c71c00342a862847e3479b1066b82114cbde34e8d848834b78fbb1b81f86f6"} Oct 30 19:52:54.422560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.420547 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" podStartSLOduration=2.420529032 podStartE2EDuration="2.420529032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:54.420362539 +0000 UTC m=+1098.549596461" watchObservedRunningTime="2026-10-30 19:52:54.420529032 +0000 UTC m=+1098.549762954" Oct 30 19:52:54.466229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.466176 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" podStartSLOduration=2.466161363 podStartE2EDuration="2.466161363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:54.462781102 +0000 UTC m=+1098.592015024" watchObservedRunningTime="2026-10-30 19:52:54.466161363 +0000 UTC m=+1098.595395284" Oct 30 19:52:54.472507 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.472469 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.815562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.810445 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:52:54.816300 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.816267 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.847104 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.847071 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9b58e9f2-openshift-default-dockercfg-bnk7n\"" Oct 30 19:52:54.864699 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.864669 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:52:54.866046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.866013 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.903233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903202 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkxg\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903237 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903370 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903313 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903373 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903395 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903519 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903557 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903594 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:54.903805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:54.903786 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005673 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005725 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005761 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005802 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005854 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.005985 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.006156 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkxg\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.006187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.006240 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.008520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.007756 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.022619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.014852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.022619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.015110 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.022619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.015870 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.029361 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.026391 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.072583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.069886 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.072583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.070624 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.097129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.097085 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.121522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.116349 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkxg\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\") pod \"e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.186521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.177917 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:55.238615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.234869 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.371583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.360620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.690049 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:52:55.690020 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36e7006_58af_4e8d_8990_dff668a88346.slice/crio-6dfa835467be4c8e40a6a669f395df9a14ed6e32bba1551826c8d3a0a20d8c6a WatchSource:0}: Error finding container 6dfa835467be4c8e40a6a669f395df9a14ed6e32bba1551826c8d3a0a20d8c6a: Status 404 returned error can't find the container with id 6dfa835467be4c8e40a6a669f395df9a14ed6e32bba1551826c8d3a0a20d8c6a Oct 30 19:52:55.690988 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.690966 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:52:55.692755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.692723 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:55.692821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.692787 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:55.692821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:55.692817 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:52:56.405427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:56.405393 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerStarted","Data":"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443"} Oct 30 19:52:56.405882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:56.405443 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerStarted","Data":"6dfa835467be4c8e40a6a669f395df9a14ed6e32bba1551826c8d3a0a20d8c6a"} Oct 30 19:52:56.482903 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:56.482849 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podStartSLOduration=2.482835923 podStartE2EDuration="2.482835923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:56.467966867 +0000 UTC m=+1100.597200789" watchObservedRunningTime="2026-10-30 19:52:56.482835923 +0000 UTC m=+1100.612069844" Oct 30 19:52:57.154890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:57.154861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.182163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:57.182122 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:57.187790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:57.187767 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:57.412242 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:57.412148 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:57.413108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:57.413093 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:52:58.133980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:58.133937 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.238950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:58.238919 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.164435 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:59.164407 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.951100 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:52:59.951059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.059762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.059718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.066723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.066694 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:53:00.071232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.071210 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.088097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.088073 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:53:00.097725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.097694 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9b58e9f2\"" Oct 30 19:53:00.097725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.097707 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9b58e9f2-dockercfg-7p6q6\"" Oct 30 19:53:00.165242 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.165208 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.165727 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.165446 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkwc\" (UniqueName: \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.303653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.290702 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.334953 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.334839 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkwc\" (UniqueName: \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.335781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.310043 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.407103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.407069 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkwc\" (UniqueName: \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\") pod \"payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.582126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.582028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.686200 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.686166 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:00.733067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.733038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.873278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.873182 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.991815 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:53:00.991786 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6092c928_28d9_45f8_9f5d_1387c5cb58b3.slice/crio-b4311f7e765302e7ad46a34aa160c78499b12c5c55d7fb623820f8f5fd4338f4 WatchSource:0}: Error finding container b4311f7e765302e7ad46a34aa160c78499b12c5c55d7fb623820f8f5fd4338f4: Status 404 returned error can't find the container with id b4311f7e765302e7ad46a34aa160c78499b12c5c55d7fb623820f8f5fd4338f4 Oct 30 19:53:00.992764 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:00.992740 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:53:01.467602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.467563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:01.476079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.476049 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" event={"ID":"6092c928-28d9-45f8-9f5d-1387c5cb58b3","Type":"ContainerStarted","Data":"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962"} Oct 30 19:53:01.476240 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.476129 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" event={"ID":"6092c928-28d9-45f8-9f5d-1387c5cb58b3","Type":"ContainerStarted","Data":"b4311f7e765302e7ad46a34aa160c78499b12c5c55d7fb623820f8f5fd4338f4"} Oct 30 19:53:01.476313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.476257 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:01.521893 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.521838 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" podStartSLOduration=1.5218225570000001 podStartE2EDuration="1.521822557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:01.518314259 +0000 UTC m=+1105.647548181" watchObservedRunningTime="2026-10-30 19:53:01.521822557 +0000 UTC m=+1105.651056477" Oct 30 19:53:01.583807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:01.583774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:03.087129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:03.087094 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.743218 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:04.743181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.869824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:04.869791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.993575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:04.992971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.447138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.447110 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:53:05.461523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.452055 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:53:05.638534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.608207 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.709769 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.704932 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:53:05.783103 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.783074 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:05.915594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.915544 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:05.916008 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.915987 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vknjg\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:05.968130 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.968034 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:05.985540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:05.981447 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.121756 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.040859 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dc2ad28e-openshift-default-dockercfg-hb6wn\"" Oct 30 19:53:06.305352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.304871 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.485524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.453639 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.511389 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.000777 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.542358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.542319 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.585843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.585761 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.586083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.586065 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.586509 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.586469 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.588045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.546645 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.588994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.546681 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:53:06.645571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.645537 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:53:06.663071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.663025 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.675762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.675617 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:06.675762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.675679 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.697904 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.697860 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.699154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.699127 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.700206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.700080 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vknjg\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.700547 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.700531 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.718482 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.701082 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.719129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.699756 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.719312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.719057 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.719529 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.719511 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.719783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.719759 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.720529 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.720476 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.722473 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.722456 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.723069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.720335 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.742804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.723854 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.743541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.741428 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.743909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.743886 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.777597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.773388 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.792522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.788763 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.810519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.804060 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vknjg\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\") pod \"e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:06.835695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.833712 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:53:06.835695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.835041 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:06.880620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.869654 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:06.880620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.870907 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:06.988981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:06.988940 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:07.042262 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.042217 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.278520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.274571 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:07.325522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.309156 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:07.325522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.310190 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:53:07.525578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.497949 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.525578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.498455 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:53:07.670730 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.670673 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:53:07.671535 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.671479 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:53:07.820586 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.750026 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:07.956624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.863182 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:53:07.956624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.952397 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:53:07.956624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:07.952443 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:53:08.070524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.064885 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.070524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.065403 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:08.070524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.065815 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:53:08.070524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.065880 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:53:08.109581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.097571 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.109581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.097971 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:53:08.172523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.171678 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Oct 30 19:53:08.172523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.171773 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:53:08.172523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.172094 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:08.172523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.172404 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:53:08.172523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.172457 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:53:08.411584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.325129 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.411584 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.352559 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:08.569729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.552177 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:53:08.569729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.565573 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.753587 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.745031 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.851542 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.770223 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.923734 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:08.922957 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:08.940298 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:53:08.940253 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335181a3_9aff_456b_8698_83486b7c7cd6.slice/crio-30b081409e5048ca0488393c6e189ddda2a4160dd86de9f5eda9302189dd5d6e WatchSource:0}: Error finding container 30b081409e5048ca0488393c6e189ddda2a4160dd86de9f5eda9302189dd5d6e: Status 404 returned error can't find the container with id 30b081409e5048ca0488393c6e189ddda2a4160dd86de9f5eda9302189dd5d6e Oct 30 19:53:09.028574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.013622 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:09.244521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.243652 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:09.297773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.297646 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.377821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.377773 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.383109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.383070 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:09.453523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.273072 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:53:09.587038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.586934 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.763165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.763115 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:09.913017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.790001 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.942525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.835701 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:09.983522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:09.970924 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.316702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.316597 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:10.379118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.378613 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:10.379118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.378688 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.670768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.661382 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.804521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.737355 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.809138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.809094 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:53:10.812524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.792877 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:53:10.812761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.812732 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.820514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.813458 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:10.897944 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.826661 4959 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-zl8sn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": context deadline exceeded" start-of-body= Oct 30 19:53:10.898203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.898172 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-zl8sn" podUID="73261ca6-68ed-4f8d-bc9e-f4eb802b75a4" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": context deadline exceeded" Oct 30 19:53:10.916851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.840645 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65" podUID="890d2c73-01dc-453a-b0d6-9db8142c9980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.917720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.875792 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:10.917951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.917920 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:10.918066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.918052 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:10.918626 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:10.756242 4959 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.529s" Oct 30 19:53:10.918815 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.918803 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:53:10.921446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.913089 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.957687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.957629 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:10.965077 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:10.958005 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.032521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.018798 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.032521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.018892 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.329335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.329234 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.529521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.501379 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.589570 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.528363 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:11.589570 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.588264 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.597626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.597582 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" event={"ID":"335181a3-9aff-456b-8698-83486b7c7cd6","Type":"ContainerStarted","Data":"30b081409e5048ca0488393c6e189ddda2a4160dd86de9f5eda9302189dd5d6e"} Oct 30 19:53:11.633540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.608600 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:53:11.738558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.697241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.738558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.732442 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.808612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.808571 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:11.823761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.810332 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.823761 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.811390 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.825317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.824572 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.904667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.904626 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:11.922679 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.922636 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:11.969121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.969077 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:11.970158 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:11.970125 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.127521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.096310 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.161683 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.161565 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:12.162003 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.161951 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.276521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.275531 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:53:12.353647 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.353603 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:12.537787 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.537701 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:53:12.558016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.557965 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.673464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.673384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.738412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.736818 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.738412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.738119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:53:12.843576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.820930 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:12.967737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.823027 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.967737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.836230 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:12.983531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.908564 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:12.983531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.930669 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:12.995752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:12.988383 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.028554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.012593 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.147523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.145861 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.377553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.295992 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:13.377553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.296073 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.377553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.296481 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.377553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.367825 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.466824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.466726 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 19:53:13.745726 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.745618 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.851808 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.851761 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.907890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.861081 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.907890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.883365 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:53:13.942118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.939417 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:13.942118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:13.939935 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.059603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.053520 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.137568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.097565 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 19:53:14.151896 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.151859 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:53:14.152587 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.152557 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:14.212514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.210232 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:14.237749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.237725 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:53:14.237969 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.237954 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:53:14.238076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.237993 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:53:14.238076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.238023 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:53:14.238837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.238818 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 19:53:14.239000 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.238985 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:53:14.240627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.240607 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" event={"ID":"335181a3-9aff-456b-8698-83486b7c7cd6","Type":"ContainerStarted","Data":"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600"} Oct 30 19:53:14.242607 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.242591 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:53:14.244117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.244097 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:53:14.531575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.531458 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" podStartSLOduration=9.531435412 podStartE2EDuration="9.531435412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:14.528769238 +0000 UTC m=+1118.658003161" watchObservedRunningTime="2026-10-30 19:53:14.531435412 +0000 UTC m=+1118.660669344" Oct 30 19:53:14.989672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.989633 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:14.995114 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:14.995091 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:15.248286 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:15.248205 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:15.249276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:15.249259 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:53:16.068208 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:16.068170 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:53:17.117336 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:17.117302 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.713068 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:17.713038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.892134 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:17.892098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:19.194858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:19.194819 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:19.651987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:19.651951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:19.793327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:19.793279 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:21.076004 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:21.075971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.662723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:22.662693 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.092125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.092095 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:53:24.096506 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.096470 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.123229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.123200 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:53:24.132943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.132919 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9b58e9f2-serving-cert\"" Oct 30 19:53:24.163977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.163941 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.275421 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.275384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.289578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.289552 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.289714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.289582 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.289714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.289669 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6t4\" (UniqueName: \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.289807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.289795 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.381825 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.381662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.392121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.392043 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.400066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.400038 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.400387 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.400364 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6t4\" (UniqueName: \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.400817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.400795 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.402104 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.402083 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.406924 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.406877 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.444967 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.444568 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.445280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.445257 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6t4\" (UniqueName: \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\") pod \"maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:24.713057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:24.713020 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:25.037427 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:53:25.037399 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf66e410_e883_48cd_adc0_fe196644269f.slice/crio-380ac73c83d496aca2730baa3e23ae5f20149da64ae709f7c6d3a0a9156eee37 WatchSource:0}: Error finding container 380ac73c83d496aca2730baa3e23ae5f20149da64ae709f7c6d3a0a9156eee37: Status 404 returned error can't find the container with id 380ac73c83d496aca2730baa3e23ae5f20149da64ae709f7c6d3a0a9156eee37 Oct 30 19:53:25.038022 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:25.037997 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:53:25.038936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:25.038918 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:53:25.418025 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:25.415822 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" event={"ID":"cf66e410-e883-48cd-adc0-fe196644269f","Type":"ContainerStarted","Data":"380ac73c83d496aca2730baa3e23ae5f20149da64ae709f7c6d3a0a9156eee37"} Oct 30 19:53:25.985628 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:25.985592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.424010 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:26.423977 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" event={"ID":"cf66e410-e883-48cd-adc0-fe196644269f","Type":"ContainerStarted","Data":"9785a47a8a3a5ac10fd0befcb58c0c41958e19e002e34ff6252365158912f05b"} Oct 30 19:53:26.424427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:26.424154 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:26.472080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:26.472021 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" podStartSLOduration=2.072641715 podStartE2EDuration="2.472007823s" totalImagesPullingTime="399.366108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:26.468071473 +0000 UTC m=+1130.597305394" watchObservedRunningTime="2026-10-30 19:53:26.472007823 +0000 UTC m=+1130.601241743" Oct 30 19:53:27.520319 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:27.520290 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:27.630880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:27.630844 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:27.742208 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:27.742173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:27.952486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:27.952456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:28.827946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:28.827910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:28.936143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:28.936108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.250287 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:29.250259 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.359880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:29.359840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.475156 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:29.475121 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.107211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:31.107178 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.048124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:32.048085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.170861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:32.170828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.437036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:32.437006 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:53:32.948985 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:32.948955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.748837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.748803 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.869115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.869087 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.921437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.921406 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:53:33.925965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.925943 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:33.947002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.946576 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c9c6d81d-dockercfg-672wp\"" Oct 30 19:53:33.949576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.949553 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:53:33.958157 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:33.958135 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c9c6d81d\"" Oct 30 19:53:34.058773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.040552 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwl7\" (UniqueName: \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.058773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.040826 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.152519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.144892 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.153082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.153051 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwl7\" (UniqueName: \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.177652 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.158903 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.304655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.294956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwl7\" (UniqueName: \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\") pod \"payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.351560 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.351466 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:53:34.369965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.369937 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:53:34.374817 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.370270 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.428558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.428531 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lf5\" (UniqueName: \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.428712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.428571 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.530407 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.530373 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lf5\" (UniqueName: \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.530583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.530416 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.531397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.531375 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.560568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.547243 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:34.652527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.649616 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lf5\" (UniqueName: \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\") pod \"payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:34.714872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:34.714839 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:35.123386 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.123340 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.147114 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:53:35.147086 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dea38e_894f_4826_9d13_2ed301c8d50f.slice/crio-fd6a56115b9f4771a865b15463a5ce9bdc50e39c3507a3443950986a739fbba0 WatchSource:0}: Error finding container fd6a56115b9f4771a865b15463a5ce9bdc50e39c3507a3443950986a739fbba0: Status 404 returned error can't find the container with id fd6a56115b9f4771a865b15463a5ce9bdc50e39c3507a3443950986a739fbba0 Oct 30 19:53:35.147826 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.147805 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:53:35.162758 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:53:35.162731 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd042ff70_6985_4cd7_a708_511103684116.slice/crio-1a1ed53a6487e7d7a63fa9925b002f48b8cad70cbd53ecc94796a0c266aba529 WatchSource:0}: Error finding container 1a1ed53a6487e7d7a63fa9925b002f48b8cad70cbd53ecc94796a0c266aba529: Status 404 returned error can't find the container with id 1a1ed53a6487e7d7a63fa9925b002f48b8cad70cbd53ecc94796a0c266aba529 Oct 30 19:53:35.164222 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.164197 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:53:35.265355 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.265211 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.486870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.486789 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" event={"ID":"d042ff70-6985-4cd7-a708-511103684116","Type":"ContainerStarted","Data":"c08d8462e5ba1a0fb4edbc4efcac18cb8798e4eded39af476345da43fd8f67b1"} Oct 30 19:53:35.486870 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.486847 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" event={"ID":"d042ff70-6985-4cd7-a708-511103684116","Type":"ContainerStarted","Data":"1a1ed53a6487e7d7a63fa9925b002f48b8cad70cbd53ecc94796a0c266aba529"} Oct 30 19:53:35.487091 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.486992 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:35.488588 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.488555 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" event={"ID":"09dea38e-894f-4826-9d13-2ed301c8d50f","Type":"ContainerStarted","Data":"7e15aecc338cff099b4b9cd46dc8661a6763bf8d96e2474e42f708d7356e33e6"} Oct 30 19:53:35.488703 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.488601 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" event={"ID":"09dea38e-894f-4826-9d13-2ed301c8d50f","Type":"ContainerStarted","Data":"fd6a56115b9f4771a865b15463a5ce9bdc50e39c3507a3443950986a739fbba0"} Oct 30 19:53:35.488754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.488723 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:35.530679 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.530634 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" podStartSLOduration=2.530619965 podStartE2EDuration="2.530619965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:35.529550778 +0000 UTC m=+1139.658784700" watchObservedRunningTime="2026-10-30 19:53:35.530619965 +0000 UTC m=+1139.659853906" Oct 30 19:53:35.573809 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.573752 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" podStartSLOduration=2.573739384 podStartE2EDuration="2.573739384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:35.571824229 +0000 UTC m=+1139.701058151" watchObservedRunningTime="2026-10-30 19:53:35.573739384 +0000 UTC m=+1139.702973304" Oct 30 19:53:35.862566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:35.857066 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:36.585239 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:36.585139 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:37.425839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:37.424757 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.354349 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:38.354310 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:39.090884 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:39.090828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.800981 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:40.800942 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:42.611601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:42.611570 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.371669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:44.371617 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:46.001520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.000597 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:46.316574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.311808 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.351515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.343226 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.649614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.607773 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:53:46.649614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.614628 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.699553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.692777 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:53:46.737739 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:46.724819 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:53:46.852522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.832708 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:46.953592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:46.951640 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.017524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.006616 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:47.074523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.054125 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:53:47.246465 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.246369 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:53:47.353067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.310673 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.354907 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.332258 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.356138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.348546 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.372755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.372727 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:53:47.373056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.373037 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:53:47.620172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.620066 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.621748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.621725 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:53:47.744698 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.744653 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.811890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.811846 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:47.841554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.836582 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 19:53:47.860032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.836619 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:47.860032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.842088 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:53:47.958526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.954888 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.958526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:47.957014 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:53:48.001980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.001946 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.054622 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.054581 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.320637 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.295306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:48.740753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.740714 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:53:48.986182 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.791819 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.986427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.824971 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 19:53:48.987171 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.825010 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:53:48.989315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.825031 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:48.989861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.825050 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:48.990266 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.936072 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:48.990705 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.936109 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:48.995131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.955799 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:48.997276 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:48.997228 4959 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.001s" Oct 30 19:53:48.997588 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.997570 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:53:48.997792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.997765 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:53:48.997936 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:48.997924 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:53:49.195185 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.108358 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 19:53:49.253598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.250723 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:53:49.253598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.246699 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:49.339375 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.339338 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:49.481977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.481950 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:53:49.509766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.509526 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 19:53:49.517790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.517762 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:49.819160 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.819077 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:53:50.046540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:49.971750 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 19:53:50.140882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.037039 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.145781 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.037083 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:53:50.146147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.037122 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:50.150274 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.037144 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:53:50.150274 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.039129 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.150274 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.110633 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:53:50.324526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.315749 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.359770 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.359716 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:50.824100 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.822591 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.933515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.930891 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:50.933515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.931274 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 19:53:50.937520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.934273 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 19:53:50.949541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:50.946568 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 19:53:51.013385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.013348 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 19:53:51.234521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.231463 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 19:53:51.242875 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.242728 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 19:53:51.244357 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.244279 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 19:53:51.319142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.288339 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 30 19:53:51.319142 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.306004 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:51.427618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.425742 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 19:53:51.427618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:51.426738 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 19:53:52.721248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:52.721208 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.483414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.483204 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" podUID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" containerName="payload-processing" containerID="cri-o://6e091f93036bd9302ffe9a64a138b8bbabd6f0d2270dd06cd6c219f7777e1ff9" gracePeriod=25 Oct 30 19:53:54.586173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.586123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.611834 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.611799 4959 generic.go:356] "Generic (PLEG): container finished" podID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" containerID="6e091f93036bd9302ffe9a64a138b8bbabd6f0d2270dd06cd6c219f7777e1ff9" exitCode=0 Oct 30 19:53:54.611972 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.611860 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" event={"ID":"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3","Type":"ContainerDied","Data":"6e091f93036bd9302ffe9a64a138b8bbabd6f0d2270dd06cd6c219f7777e1ff9"} Oct 30 19:53:54.847797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.847743 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" podUID="981f87e9-f3a5-4f9c-8a54-59549a463c62" containerName="payload-pre-processing" containerID="cri-o://2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395" gracePeriod=25 Oct 30 19:53:54.994405 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:54.994380 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:53:55.054734 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.054700 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\") pod \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " Oct 30 19:53:55.057608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.054766 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\" (UniqueName: \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\") pod \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\" (UID: \"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3\") " Oct 30 19:53:55.057608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.055878 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume" pod "f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" (UID: "f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:55.058656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.058633 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h" pod "f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" (UID: "f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3"). InnerVolumeSpecName "kube-api-access-zhz5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:55.157397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.157334 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhz5h\" (UniqueName: \"kubernetes.io/projected/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-kube-api-access-zhz5h\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:53:55.157397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.157364 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:53:55.168674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.168649 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:53:55.258862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.258830 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\") pod \"981f87e9-f3a5-4f9c-8a54-59549a463c62\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " Oct 30 19:53:55.259110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.259076 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\" (UniqueName: \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\") pod \"981f87e9-f3a5-4f9c-8a54-59549a463c62\" (UID: \"981f87e9-f3a5-4f9c-8a54-59549a463c62\") " Oct 30 19:53:55.259667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.259632 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume" pod "981f87e9-f3a5-4f9c-8a54-59549a463c62" (UID: "981f87e9-f3a5-4f9c-8a54-59549a463c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:55.259822 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.259754 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981f87e9-f3a5-4f9c-8a54-59549a463c62-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:53:55.262198 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.262175 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn" pod "981f87e9-f3a5-4f9c-8a54-59549a463c62" (UID: "981f87e9-f3a5-4f9c-8a54-59549a463c62"). InnerVolumeSpecName "kube-api-access-lzqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:55.361882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.361806 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzqbn\" (UniqueName: \"kubernetes.io/projected/981f87e9-f3a5-4f9c-8a54-59549a463c62-kube-api-access-lzqbn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:53:55.620363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.620274 4959 generic.go:356] "Generic (PLEG): container finished" podID="981f87e9-f3a5-4f9c-8a54-59549a463c62" containerID="2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395" exitCode=0 Oct 30 19:53:55.620363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.620320 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" Oct 30 19:53:55.620882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.620364 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" event={"ID":"981f87e9-f3a5-4f9c-8a54-59549a463c62","Type":"ContainerDied","Data":"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395"} Oct 30 19:53:55.620882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.620419 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg" event={"ID":"981f87e9-f3a5-4f9c-8a54-59549a463c62","Type":"ContainerDied","Data":"f00038ba1138873c0a36b9128f5e11e1716adba86efaee8e0dfd58b4b8ad6ca4"} Oct 30 19:53:55.620882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.620451 4959 scope.go:122] "RemoveContainer" containerID="2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395" Oct 30 19:53:55.622203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.622181 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" Oct 30 19:53:55.622313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.622186 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m" event={"ID":"f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3","Type":"ContainerDied","Data":"09c71c00342a862847e3479b1066b82114cbde34e8d848834b78fbb1b81f86f6"} Oct 30 19:53:55.634387 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.634340 4959 scope.go:122] "RemoveContainer" containerID="2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395" Oct 30 19:53:55.635940 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:55.635906 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395\": container with ID starting with 2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395 not found: ID does not exist" containerID="2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395" Oct 30 19:53:55.636021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.635948 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395"} err="failed to get container status \"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395\": rpc error: code = NotFound desc = could not find container \"2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395\": container with ID starting with 2644b6dd2a0f96f869080701de219367bbb8b112e59118e3c1f8c059af689395 not found: ID does not exist" Oct 30 19:53:55.636021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.635973 4959 scope.go:122] "RemoveContainer" containerID="6e091f93036bd9302ffe9a64a138b8bbabd6f0d2270dd06cd6c219f7777e1ff9" Oct 30 19:53:55.680444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.680412 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:53:55.683642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.683620 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffc83e66-5c66f649b-zdm6m"] Oct 30 19:53:55.716154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.716052 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:53:55.722166 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:55.722136 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ffc83e66-7b78594d67-r5xsg"] Oct 30 19:53:56.002366 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.002330 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981f87e9-f3a5-4f9c-8a54-59549a463c62" path="/var/lib/kubelet/pods/981f87e9-f3a5-4f9c-8a54-59549a463c62/volumes" Oct 30 19:53:56.002937 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.002915 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" path="/var/lib/kubelet/pods/f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3/volumes" Oct 30 19:53:56.550845 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.550812 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:53:56.551530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551474 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" containerName="payload-processing" Oct 30 19:53:56.551530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551505 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" containerName="payload-processing" Oct 30 19:53:56.551530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551521 4959 cpu_manager.go:414] "containerMap: removing container" podUID="981f87e9-f3a5-4f9c-8a54-59549a463c62" containerName="payload-pre-processing" Oct 30 19:53:56.551530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551528 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981f87e9-f3a5-4f9c-8a54-59549a463c62" containerName="payload-pre-processing" Oct 30 19:53:56.551829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551703 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25c63a8-59e3-4d43-9f92-c5d0c32dc1e3" containerName="payload-processing" Oct 30 19:53:56.551829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.551715 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="981f87e9-f3a5-4f9c-8a54-59549a463c62" containerName="payload-pre-processing" Oct 30 19:53:56.556999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.556977 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.580827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.580789 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b55z\" (UniqueName: \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.581053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.581032 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.581149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.581133 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.581214 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.581199 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.591199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.587011 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c9c6d81d-serving-cert\"" Oct 30 19:53:56.593212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.593184 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:53:56.647233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.647204 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.683112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.683074 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.683304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.683172 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.683304 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.683219 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.683428 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:56.683334 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-serving-cert: secret "maas-api-e2e-derive-c9c6d81d-serving-cert" not found Oct 30 19:53:56.683428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.683353 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b55z\" (UniqueName: \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.683428 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:53:56.683404 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls podName:754e1383-7314-4d17-bf79-9c8c46661db8 nodeName:}" failed. No retries permitted until 2026-10-30 19:53:57.183383654 +0000 UTC m=+1161.312617554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls") pod "maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" (UID: "754e1383-7314-4d17-bf79-9c8c46661db8") : secret "maas-api-e2e-derive-c9c6d81d-serving-cert" not found Oct 30 19:53:56.683940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.683918 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.686169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.686142 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.706911 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.706883 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b55z\" (UniqueName: \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:56.784562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.772306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.895561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:56.893229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:57.199830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:57.199743 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:57.202431 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:57.202406 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") pod \"maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:57.473328 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:57.473241 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:57.849581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:57.849013 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:53:58.460632 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.460598 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:53:58.460804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.460671 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:58.662915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.662883 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" event={"ID":"754e1383-7314-4d17-bf79-9c8c46661db8","Type":"ContainerStarted","Data":"d11c155e1a9b8614ff9542aa4c943d2a909730671c83c44cf25cd492cd22ec38"} Oct 30 19:53:58.663098 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.662946 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:53:58.663098 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.662968 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" event={"ID":"754e1383-7314-4d17-bf79-9c8c46661db8","Type":"ContainerStarted","Data":"6b6c1c5e571b786b335b26390d40c1bc2a17eebd0e355590c425b3edbc739d13"} Oct 30 19:53:58.681129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.681098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:58.711480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:58.711428 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" podStartSLOduration=2.290141648 podStartE2EDuration="2.711411661s" totalImagesPullingTime="421.270013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:58.709454956 +0000 UTC m=+1162.838688878" watchObservedRunningTime="2026-10-30 19:53:58.711411661 +0000 UTC m=+1162.840645583" Oct 30 19:53:59.671232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.671167 4959 generic.go:356] "Generic (PLEG): container finished" podID="a36e7006-58af-4e8d-8990-dff668a88346" containerID="cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443" exitCode=0 Oct 30 19:53:59.671686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.671236 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerDied","Data":"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443"} Oct 30 19:53:59.671686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.671613 4959 scope.go:122] "RemoveContainer" containerID="cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443" Oct 30 19:53:59.673063 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.673032 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:59.673186 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.673116 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:59.673186 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.673164 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:53:59.952608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:53:59.952562 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:00.586414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:00.586255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:00.712546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:00.710205 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerStarted","Data":"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156"} Oct 30 19:54:00.734519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:00.734008 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:00.928289 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:00.928255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.006915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.006871 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:54:01.078784 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.078613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.183802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.183699 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:01.184220 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.184194 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:01.184305 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.184261 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:01.200185 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.200152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.681413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.681381 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.785707 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.785673 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.960571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:01.960025 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.202622 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:02.202586 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:02.202821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:02.202660 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:02.476333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:02.476286 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.771881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:02.771796 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:03.122576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:03.114032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:03.186519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:03.184798 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:03.186519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:03.184858 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:04.182361 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:04.182071 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:04.182361 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:04.182122 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:04.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:04.678070 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:54:05.139901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:05.139868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:05.182692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:05.182655 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:05.183476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:05.182725 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:05.632124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:05.632036 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:05.761970 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:05.761935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.008582 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.008538 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" podUID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" containerName="payload-processing" containerID="cri-o://817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962" gracePeriod=25 Oct 30 19:54:06.182842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.182352 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:06.182842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.182412 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:06.335837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.335813 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:54:06.479534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.468405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.497518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.495434 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\" (UniqueName: \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\") pod \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " Oct 30 19:54:06.566603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.560063 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc" pod "6092c928-28d9-45f8-9f5d-1387c5cb58b3" (UID: "6092c928-28d9-45f8-9f5d-1387c5cb58b3"). InnerVolumeSpecName "kube-api-access-6fkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:06.566603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.560258 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\") pod \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\" (UID: \"6092c928-28d9-45f8-9f5d-1387c5cb58b3\") " Oct 30 19:54:06.573563 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.568090 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fkwc\" (UniqueName: \"kubernetes.io/projected/6092c928-28d9-45f8-9f5d-1387c5cb58b3-kube-api-access-6fkwc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:06.603522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.573928 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume" pod "6092c928-28d9-45f8-9f5d-1387c5cb58b3" (UID: "6092c928-28d9-45f8-9f5d-1387c5cb58b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:06.673921 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.673880 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6092c928-28d9-45f8-9f5d-1387c5cb58b3-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:06.855608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.830120 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.969592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.962915 4959 generic.go:356] "Generic (PLEG): container finished" podID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" containerID="817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962" exitCode=0 Oct 30 19:54:06.969592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.963018 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" event={"ID":"6092c928-28d9-45f8-9f5d-1387c5cb58b3","Type":"ContainerDied","Data":"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962"} Oct 30 19:54:06.969592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.963072 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" event={"ID":"6092c928-28d9-45f8-9f5d-1387c5cb58b3","Type":"ContainerDied","Data":"b4311f7e765302e7ad46a34aa160c78499b12c5c55d7fb623820f8f5fd4338f4"} Oct 30 19:54:06.969592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.963106 4959 scope.go:122] "RemoveContainer" containerID="817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962" Oct 30 19:54:07.005520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:06.971697 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw" Oct 30 19:54:07.124445 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.119952 4959 scope.go:122] "RemoveContainer" containerID="817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962" Oct 30 19:54:07.135762 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:07.135716 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962\": container with ID starting with 817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962 not found: ID does not exist" containerID="817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962" Oct 30 19:54:07.136038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.136008 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962"} err="failed to get container status \"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962\": rpc error: code = NotFound desc = could not find container \"817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962\": container with ID starting with 817718c41b2384edfdfde9d2cb6014f3ac635c67726fc284255a70e73e557962 not found: ID does not exist" Oct 30 19:54:07.191530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.188865 4959 patch_prober.go:29] interesting pod/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:54:07.191530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.188928 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:54:07.256608 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.236607 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:07.412583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.412553 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:54:07.513224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:07.513192 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9b58e9f2-57c57bd858-fnnfw"] Oct 30 19:54:08.000400 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:08.000366 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" path="/var/lib/kubelet/pods/6092c928-28d9-45f8-9f5d-1387c5cb58b3/volumes" Oct 30 19:54:08.187241 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:08.187213 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:08.187649 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:08.187633 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:08.188246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:08.188230 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:09.055646 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.055610 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 19:54:09.056320 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.056304 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" containerName="payload-processing" Oct 30 19:54:09.056359 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.056321 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" containerName="payload-processing" Oct 30 19:54:09.056456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.056446 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6092c928-28d9-45f8-9f5d-1387c5cb58b3" containerName="payload-processing" Oct 30 19:54:09.061659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.061633 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.097232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.097201 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 19:54:09.107888 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.107822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.118358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118324 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118456 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118554 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118486 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118626 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118715 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118806 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.118952 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.118937 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.119259 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.119240 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.119352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.119312 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2hd\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221453 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221594 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221564 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221632 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2hd\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221792 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221876 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221909 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.221984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.221944 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.222272 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.222020 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.222272 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.222067 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.223347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.223318 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.223885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.223629 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.223885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.223665 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.223885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.223851 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.224229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.224216 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.263615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.246118 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.263615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.249848 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.263615 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.258078 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.280562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.276636 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2hd\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-gqnwx\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.377053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.376976 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:09.715012 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:54:09.714978 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af5edd2_6de4_49c1_b711_48d353cab69d.slice/crio-fa882d34cf9c479dfe8151aec9fa5db5a3b6c39d1f5799bf46f086f657ffa70e WatchSource:0}: Error finding container fa882d34cf9c479dfe8151aec9fa5db5a3b6c39d1f5799bf46f086f657ffa70e: Status 404 returned error can't find the container with id fa882d34cf9c479dfe8151aec9fa5db5a3b6c39d1f5799bf46f086f657ffa70e Oct 30 19:54:09.715294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.715271 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 19:54:09.717751 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.717718 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:09.717843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.717783 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:09.717843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.717813 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:09.988023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.987915 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" event={"ID":"8af5edd2-6de4-49c1-b711-48d353cab69d","Type":"ContainerStarted","Data":"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45"} Oct 30 19:54:09.988023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:09.987986 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" event={"ID":"8af5edd2-6de4-49c1-b711-48d353cab69d","Type":"ContainerStarted","Data":"fa882d34cf9c479dfe8151aec9fa5db5a3b6c39d1f5799bf46f086f657ffa70e"} Oct 30 19:54:10.378012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.377917 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:10.379085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.379063 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:10.379169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.379112 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:10.780522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.762363 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podStartSLOduration=1.76233929 podStartE2EDuration="1.76233929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:10.035452102 +0000 UTC m=+1174.164686031" watchObservedRunningTime="2026-10-30 19:54:10.76233929 +0000 UTC m=+1174.891573211" Oct 30 19:54:10.781061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.781030 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:54:10.781456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.781416 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" podUID="cf66e410-e883-48cd-adc0-fe196644269f" containerName="maas-api" containerID="cri-o://9785a47a8a3a5ac10fd0befcb58c0c41958e19e002e34ff6252365158912f05b" gracePeriod=30 Oct 30 19:54:10.814753 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:10.814202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:11.397334 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.391644 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:11.397334 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.391749 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:11.400865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.399744 4959 generic.go:356] "Generic (PLEG): container finished" podID="cf66e410-e883-48cd-adc0-fe196644269f" containerID="9785a47a8a3a5ac10fd0befcb58c0c41958e19e002e34ff6252365158912f05b" exitCode=0 Oct 30 19:54:11.400865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.400578 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" event={"ID":"cf66e410-e883-48cd-adc0-fe196644269f","Type":"ContainerDied","Data":"9785a47a8a3a5ac10fd0befcb58c0c41958e19e002e34ff6252365158912f05b"} Oct 30 19:54:11.408387 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.408310 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:54:11.523549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.523516 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\") pod \"cf66e410-e883-48cd-adc0-fe196644269f\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " Oct 30 19:54:11.524911 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.524885 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\") pod \"cf66e410-e883-48cd-adc0-fe196644269f\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " Oct 30 19:54:11.531852 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.531822 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\" (UniqueName: \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\") pod \"cf66e410-e883-48cd-adc0-fe196644269f\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " Oct 30 19:54:11.532612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.532588 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\") pod \"cf66e410-e883-48cd-adc0-fe196644269f\" (UID: \"cf66e410-e883-48cd-adc0-fe196644269f\") " Oct 30 19:54:11.553521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.524212 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca" pod "cf66e410-e883-48cd-adc0-fe196644269f" (UID: "cf66e410-e883-48cd-adc0-fe196644269f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:11.609570 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.600663 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4" pod "cf66e410-e883-48cd-adc0-fe196644269f" (UID: "cf66e410-e883-48cd-adc0-fe196644269f"). InnerVolumeSpecName "kube-api-access-hb6t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:11.651524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.628101 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls" pod "cf66e410-e883-48cd-adc0-fe196644269f" (UID: "cf66e410-e883-48cd-adc0-fe196644269f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:11.651524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.639280 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:11.651524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.639307 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf66e410-e883-48cd-adc0-fe196644269f-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:11.651524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.639324 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6t4\" (UniqueName: \"kubernetes.io/projected/cf66e410-e883-48cd-adc0-fe196644269f-kube-api-access-hb6t4\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:11.794517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.794304 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs" pod "cf66e410-e883-48cd-adc0-fe196644269f" (UID: "cf66e410-e883-48cd-adc0-fe196644269f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:11.845117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.839837 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf66e410-e883-48cd-adc0-fe196644269f-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:11.988892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:11.988858 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.117215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.117181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.247135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.247053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.408521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.403098 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:12.408521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.403175 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:12.442649 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.442606 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" event={"ID":"cf66e410-e883-48cd-adc0-fe196644269f","Type":"ContainerDied","Data":"380ac73c83d496aca2730baa3e23ae5f20149da64ae709f7c6d3a0a9156eee37"} Oct 30 19:54:12.442848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.442673 4959 scope.go:122] "RemoveContainer" containerID="9785a47a8a3a5ac10fd0befcb58c0c41958e19e002e34ff6252365158912f05b" Oct 30 19:54:12.442848 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.442628 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn" Oct 30 19:54:12.511894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.511816 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:54:12.524691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:12.524663 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9b58e9f2-6f779fdc69-959kn"] Oct 30 19:54:13.377994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:13.377963 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:13.378215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:13.378019 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:13.695987 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:13.695955 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:54:13.696468 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:13.696180 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" containerID="cri-o://675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156" gracePeriod=30 Oct 30 19:54:13.710306 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:13.710280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.065566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.064999 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf66e410-e883-48cd-adc0-fe196644269f" path="/var/lib/kubelet/pods/cf66e410-e883-48cd-adc0-fe196644269f/volumes" Oct 30 19:54:14.378046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.377948 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:14.378046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.378009 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:14.489945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.489907 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.626622 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.623978 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.741218 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.741189 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.873567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:14.871062 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.377968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:15.377928 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:15.378154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:15.377991 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:16.081817 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:16.081779 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:54:16.311080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:16.311040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.380114 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:16.380026 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:16.380362 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:16.380332 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:17.378246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:17.378209 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:17.378752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:17.378271 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 19:54:17.951515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:17.951470 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:18.752635 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:18.752567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:18.890139 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:18.885125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:19.070110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.069093 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:19.071252 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.070851 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:19.075530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.071703 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 19:54:19.168431 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.168407 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:19.213086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213060 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213265 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213242 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213347 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213421 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213405 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213462 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213445 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213480 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213555 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213635 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213622 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.213726 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213710 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\") pod \"a36e7006-58af-4e8d-8990-dff668a88346\" (UID: \"a36e7006-58af-4e8d-8990-dff668a88346\") " Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213348 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.214770 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.214778 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213663 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.214794 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.214828 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.213879 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:19.219717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.218013 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "kube-api-access-8qkxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:19.219717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.218241 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:19.219717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.218248 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:19.221764 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.220249 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo" pod "a36e7006-58af-4e8d-8990-dff668a88346" (UID: "a36e7006-58af-4e8d-8990-dff668a88346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316206 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a36e7006-58af-4e8d-8990-dff668a88346-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316230 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316240 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkxg\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-kube-api-access-8qkxg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316249 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316257 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a36e7006-58af-4e8d-8990-dff668a88346-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316265 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316275 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316273 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a36e7006-58af-4e8d-8990-dff668a88346-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.316621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.316283 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a36e7006-58af-4e8d-8990-dff668a88346-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:19.562638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.553736 4959 generic.go:356] "Generic (PLEG): container finished" podID="a36e7006-58af-4e8d-8990-dff668a88346" containerID="675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156" exitCode=0 Oct 30 19:54:19.562638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.554534 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" Oct 30 19:54:19.562638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.554837 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerDied","Data":"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156"} Oct 30 19:54:19.562638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.554882 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg" event={"ID":"a36e7006-58af-4e8d-8990-dff668a88346","Type":"ContainerDied","Data":"6dfa835467be4c8e40a6a669f395df9a14ed6e32bba1551826c8d3a0a20d8c6a"} Oct 30 19:54:19.562638 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.554910 4959 scope.go:122] "RemoveContainer" containerID="675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156" Oct 30 19:54:19.665982 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.664779 4959 scope.go:122] "RemoveContainer" containerID="cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443" Oct 30 19:54:19.681453 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.681409 4959 scope.go:122] "RemoveContainer" containerID="675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156" Oct 30 19:54:19.681822 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:19.681801 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156\": container with ID starting with 675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156 not found: ID does not exist" containerID="675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156" Oct 30 19:54:19.681922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.681835 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156"} err="failed to get container status \"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156\": rpc error: code = NotFound desc = could not find container \"675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156\": container with ID starting with 675f56774e18d01a7a9979662a02dec037cf2b81747a880b5d8488f9d3175156 not found: ID does not exist" Oct 30 19:54:19.681922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.681863 4959 scope.go:122] "RemoveContainer" containerID="cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443" Oct 30 19:54:19.682243 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:19.682195 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443\": container with ID starting with cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443 not found: ID does not exist" containerID="cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443" Oct 30 19:54:19.682310 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.682253 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443"} err="failed to get container status \"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443\": rpc error: code = NotFound desc = could not find container \"cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443\": container with ID starting with cb9300a96b46d8d08cbc0eac5c3d53b05ec14daecaa385fd078d83c62f65d443 not found: ID does not exist" Oct 30 19:54:19.721962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.721933 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:54:19.726964 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.726935 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9b58e9f2-openshift-default-d77745f89-ltjdg"] Oct 30 19:54:19.780798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:19.780772 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:20.040573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:20.031625 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36e7006-58af-4e8d-8990-dff668a88346" path="/var/lib/kubelet/pods/a36e7006-58af-4e8d-8990-dff668a88346/volumes" Oct 30 19:54:21.297427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:21.297377 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.045125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.009969 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.197237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.197199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.327578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.325703 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.863390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.863355 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:54:22.863625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.863594 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" podUID="335181a3-9aff-456b-8698-83486b7c7cd6" containerName="istio-proxy" containerID="cri-o://890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600" gracePeriod=30 Oct 30 19:54:22.914830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:22.914799 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.893950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.893912 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:54:23.894694 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894674 4959 cpu_manager.go:414] "containerMap: removing container" podUID="cf66e410-e883-48cd-adc0-fe196644269f" containerName="maas-api" Oct 30 19:54:23.894800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894695 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf66e410-e883-48cd-adc0-fe196644269f" containerName="maas-api" Oct 30 19:54:23.894800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894733 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:23.894800 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894740 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:23.894924 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894910 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf66e410-e883-48cd-adc0-fe196644269f" containerName="maas-api" Oct 30 19:54:23.894960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894937 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:23.894960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.894952 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:23.898556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.898538 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:23.935255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.927795 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-75082b11-serving-cert\"" Oct 30 19:54:23.940257 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.940230 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:54:23.971846 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.971802 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.985559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.984866 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:23.985559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.985048 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mxs\" (UniqueName: \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:23.985559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.985160 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:23.985559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:23.985267 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.085063 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.085022 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 19:54:24.086152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.086122 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:24.086152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.086153 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36e7006-58af-4e8d-8990-dff668a88346" containerName="istio-proxy" Oct 30 19:54:24.087802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.087771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.089081 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.089049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.089258 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.089238 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mxs\" (UniqueName: \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.089374 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.089346 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.089559 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.089541 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.090429 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.090406 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.093096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.093054 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.161261 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.120342 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.161261 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.123228 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 19:54:24.161261 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.123355 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.199545 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.182815 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mxs\" (UniqueName: \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\") pod \"maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.230919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.230652 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:24.334235 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.334199 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.334892 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.334660 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.334937 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.334996 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.335033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.335087 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.335146 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.335177 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.335550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.335219 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnx59\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.436665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.436583 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.437009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.436987 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.437135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.437120 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.437479 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.437461 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.437777 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.437757 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.437834 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.437819 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.438442 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.438422 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.438558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.438541 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.438824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.438808 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.438890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.438876 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.439225 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.439207 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.439298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.439257 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnx59\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.439679 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.439413 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.439949 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.439931 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.441532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.441482 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.441811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.441792 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.460295 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.460262 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.460413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.460360 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnx59\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\") pod \"maas-default-gateway-openshift-default-75bf867b6c-fh6gc\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.556580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.556525 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.667605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.667568 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:24.718659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:24.697340 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:54:25.444482 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.444447 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 19:54:25.460021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.459061 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:25.460021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.459151 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:25.460021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.459197 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:54:25.688991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.688899 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" event={"ID":"e3c04781-2619-4bc2-af4e-6c873bd49547","Type":"ContainerStarted","Data":"c654f0f934a90fd328ab9739463b7cc8e26b218fb44e380e0f38f3e98eaddfba"} Oct 30 19:54:25.694006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.693935 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" event={"ID":"3916ece0-cabe-44a6-9511-f538c677a751","Type":"ContainerStarted","Data":"bef599b143aa7df92b783a50398d431f8b969adf690d09a3ec7914ec72d63a34"} Oct 30 19:54:25.884521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:25.884463 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:26.471210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.471177 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:26.730845 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.730745 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" event={"ID":"3916ece0-cabe-44a6-9511-f538c677a751","Type":"ContainerStarted","Data":"482c650391c19a19c02b2aead89a610e104b7ebedcfa797e5693846fac679fde"} Oct 30 19:54:26.731189 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.731164 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:26.732926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.732886 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" event={"ID":"e3c04781-2619-4bc2-af4e-6c873bd49547","Type":"ContainerStarted","Data":"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a"} Oct 30 19:54:26.785990 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.785924 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" podStartSLOduration=3.082894229 podStartE2EDuration="3.785904077s" totalImagesPullingTime="703.009848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:26.774948642 +0000 UTC m=+1190.904182564" watchObservedRunningTime="2026-10-30 19:54:26.785904077 +0000 UTC m=+1190.915138000" Oct 30 19:54:26.825977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:26.825913 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podStartSLOduration=2.825893562 podStartE2EDuration="2.825893562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:26.822357936 +0000 UTC m=+1190.951591884" watchObservedRunningTime="2026-10-30 19:54:26.825893562 +0000 UTC m=+1190.955127483" Oct 30 19:54:27.686518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:27.674382 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:27.686518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:27.674667 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:27.686518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:27.674734 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:28.292419 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.292350 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:54:28.443030 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.442993 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443220 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443046 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443220 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443195 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443334 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443225 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443387 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443360 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:28.443448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443369 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443529 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443482 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443589 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443573 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443683 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443664 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443720 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\") pod \"335181a3-9aff-456b-8698-83486b7c7cd6\" (UID: \"335181a3-9aff-456b-8698-83486b7c7cd6\") " Oct 30 19:54:28.443773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443575 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:28.443773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443585 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:28.443958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.443934 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:28.444026 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444005 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:28.444539 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444480 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/335181a3-9aff-456b-8698-83486b7c7cd6-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.444648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444543 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.444648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444555 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.444648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444566 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.444648 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.444579 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.446045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.445806 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:28.446045 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.445849 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:28.446206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.446134 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "kube-api-access-vknjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:28.446206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.446147 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token" pod "335181a3-9aff-456b-8698-83486b7c7cd6" (UID: "335181a3-9aff-456b-8698-83486b7c7cd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:28.546097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.546017 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.546097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.546053 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/335181a3-9aff-456b-8698-83486b7c7cd6-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.546097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.546068 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vknjg\" (UniqueName: \"kubernetes.io/projected/335181a3-9aff-456b-8698-83486b7c7cd6-kube-api-access-vknjg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.546097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.546083 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/335181a3-9aff-456b-8698-83486b7c7cd6-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:28.673117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.673072 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:28.673296 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.673148 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:28.829801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.829683 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.901789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.901757 4959 generic.go:356] "Generic (PLEG): container finished" podID="335181a3-9aff-456b-8698-83486b7c7cd6" containerID="890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600" exitCode=0 Oct 30 19:54:28.901965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.901819 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" Oct 30 19:54:28.901965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.901839 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" event={"ID":"335181a3-9aff-456b-8698-83486b7c7cd6","Type":"ContainerDied","Data":"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600"} Oct 30 19:54:28.901965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.901900 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl" event={"ID":"335181a3-9aff-456b-8698-83486b7c7cd6","Type":"ContainerDied","Data":"30b081409e5048ca0488393c6e189ddda2a4160dd86de9f5eda9302189dd5d6e"} Oct 30 19:54:28.901965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.901931 4959 scope.go:122] "RemoveContainer" containerID="890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600" Oct 30 19:54:28.916830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.916255 4959 scope.go:122] "RemoveContainer" containerID="890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600" Oct 30 19:54:28.916830 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:28.916614 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600\": container with ID starting with 890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600 not found: ID does not exist" containerID="890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600" Oct 30 19:54:28.916830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.916648 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600"} err="failed to get container status \"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600\": rpc error: code = NotFound desc = could not find container \"890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600\": container with ID starting with 890614b475d99b8fedf9cfb109aad984e36b9438685c9b8a5f0757b808416600 not found: ID does not exist" Oct 30 19:54:28.973556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.973457 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.977351 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.977306 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:54:28.982224 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:28.982187 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dc2ad28e-openshift-default-98697487f-6p4nl"] Oct 30 19:54:29.087686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:29.087601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.269521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:29.268060 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.673023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:29.672981 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:29.673243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:29.673048 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:29.998263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:29.998179 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335181a3-9aff-456b-8698-83486b7c7cd6" path="/var/lib/kubelet/pods/335181a3-9aff-456b-8698-83486b7c7cd6/volumes" Oct 30 19:54:30.673583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.671932 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:30.673583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.671995 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:30.693178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.693054 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:30.804167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.804133 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:54:30.804880 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.804865 4959 cpu_manager.go:414] "containerMap: removing container" podUID="335181a3-9aff-456b-8698-83486b7c7cd6" containerName="istio-proxy" Oct 30 19:54:30.804920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.804883 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335181a3-9aff-456b-8698-83486b7c7cd6" containerName="istio-proxy" Oct 30 19:54:30.805020 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.805011 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="335181a3-9aff-456b-8698-83486b7c7cd6" containerName="istio-proxy" Oct 30 19:54:30.810651 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.810632 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:30.828399 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.828375 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:54:30.836771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.836745 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c53b7887\"" Oct 30 19:54:30.892019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.891987 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:54:30.901428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.901402 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:54:30.901619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.901558 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:30.915440 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.915418 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdntk\" (UniqueName: \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:30.920591 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.915825 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:30.920591 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:30.917885 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c53b7887-dockercfg-wtgrr\"" Oct 30 19:54:31.071602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.068831 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:31.071602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.069097 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdntk\" (UniqueName: \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:31.071602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.069176 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk69w\" (UniqueName: \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.071602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.069350 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.071602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.070539 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:31.192041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.174867 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk69w\" (UniqueName: \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.192041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.175013 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.192539 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.192367 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.383581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.373365 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdntk\" (UniqueName: \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\") pod \"payload-processing-e2e-mt-c53b7887-dfd595754-d4ths\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:31.404569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.400356 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk69w\" (UniqueName: \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\") pod \"payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.426874 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.426847 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:31.531521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.531351 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:31.671634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.671589 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.671835 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.671704 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:31.697611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.697574 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:31.940908 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:54:31.940878 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d151067_78ac_4601_8f06_9cb7c40d94d5.slice/crio-1631d5cfa41c85c96e3fd97b83ab602d22e88db922c76035143396b5b0c213a4 WatchSource:0}: Error finding container 1631d5cfa41c85c96e3fd97b83ab602d22e88db922c76035143396b5b0c213a4: Status 404 returned error can't find the container with id 1631d5cfa41c85c96e3fd97b83ab602d22e88db922c76035143396b5b0c213a4 Oct 30 19:54:31.942002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:31.941978 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:54:32.000562 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:54:32.000531 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eea479_871e_48ab_980b_67bbdc837f71.slice/crio-c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8 WatchSource:0}: Error finding container c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8: Status 404 returned error can't find the container with id c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8 Oct 30 19:54:32.004227 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.004203 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:54:32.325550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.325438 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" event={"ID":"e2eea479-871e-48ab-980b-67bbdc837f71","Type":"ContainerStarted","Data":"a38c80072eb4bb2c0cefa8709fdef0ef43819abee8093c7f61379f52126ed98f"} Oct 30 19:54:32.325550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.325531 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" event={"ID":"e2eea479-871e-48ab-980b-67bbdc837f71","Type":"ContainerStarted","Data":"c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8"} Oct 30 19:54:32.326068 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.325560 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:32.327217 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.327193 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" event={"ID":"2d151067-78ac-4601-8f06-9cb7c40d94d5","Type":"ContainerStarted","Data":"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868"} Oct 30 19:54:32.327352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.327232 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" event={"ID":"2d151067-78ac-4601-8f06-9cb7c40d94d5","Type":"ContainerStarted","Data":"1631d5cfa41c85c96e3fd97b83ab602d22e88db922c76035143396b5b0c213a4"} Oct 30 19:54:32.327352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.327275 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:32.371603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.371541 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" podStartSLOduration=2.3715206970000002 podStartE2EDuration="2.371520697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:32.369864327 +0000 UTC m=+1196.499098250" watchObservedRunningTime="2026-10-30 19:54:32.371520697 +0000 UTC m=+1196.500754620" Oct 30 19:54:32.420883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.420830 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" podStartSLOduration=2.420817767 podStartE2EDuration="2.420817767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:32.419722388 +0000 UTC m=+1196.548956311" watchObservedRunningTime="2026-10-30 19:54:32.420817767 +0000 UTC m=+1196.550051704" Oct 30 19:54:32.672522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.670841 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:32.672522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.670903 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:32.901087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.901059 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:54:32.932431 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.932366 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:54:32.981750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:32.981719 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:54:33.114721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:33.114688 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.235395 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:33.235309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.375069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:33.375035 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.670882 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:33.670851 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.671033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:33.670926 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:34.670601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:34.670557 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-fh6gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:54:34.671129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:34.670611 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:54:35.076067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:35.075968 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:35.802000 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:35.801962 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:35.802652 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:35.802289 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:35.803437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:35.803384 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 19:54:36.126091 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:36.126019 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:54:36.132301 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:36.132277 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:54:36.786220 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:36.786190 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.466191 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:37.466158 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.933150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:37.933111 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" podUID="d042ff70-6985-4cd7-a708-511103684116" containerName="payload-processing" containerID="cri-o://c08d8462e5ba1a0fb4edbc4efcac18cb8798e4eded39af476345da43fd8f67b1" gracePeriod=25 Oct 30 19:54:37.982308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:37.982245 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" podUID="09dea38e-894f-4826-9d13-2ed301c8d50f" containerName="payload-pre-processing" containerID="cri-o://7e15aecc338cff099b4b9cd46dc8661a6763bf8d96e2474e42f708d7356e33e6" gracePeriod=25 Oct 30 19:54:38.071419 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.071386 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:54:38.071881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.071845 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" podUID="754e1383-7314-4d17-bf79-9c8c46661db8" containerName="maas-api" containerID="cri-o://d11c155e1a9b8614ff9542aa4c943d2a909730671c83c44cf25cd492cd22ec38" gracePeriod=30 Oct 30 19:54:38.119583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.119481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.542907 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.542819 4959 generic.go:356] "Generic (PLEG): container finished" podID="754e1383-7314-4d17-bf79-9c8c46661db8" containerID="d11c155e1a9b8614ff9542aa4c943d2a909730671c83c44cf25cd492cd22ec38" exitCode=0 Oct 30 19:54:38.587271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.543798 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" event={"ID":"754e1383-7314-4d17-bf79-9c8c46661db8","Type":"ContainerDied","Data":"d11c155e1a9b8614ff9542aa4c943d2a909730671c83c44cf25cd492cd22ec38"} Oct 30 19:54:38.587808 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.579945 4959 generic.go:356] "Generic (PLEG): container finished" podID="d042ff70-6985-4cd7-a708-511103684116" containerID="c08d8462e5ba1a0fb4edbc4efcac18cb8798e4eded39af476345da43fd8f67b1" exitCode=0 Oct 30 19:54:38.588231 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.588030 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" event={"ID":"d042ff70-6985-4cd7-a708-511103684116","Type":"ContainerDied","Data":"c08d8462e5ba1a0fb4edbc4efcac18cb8798e4eded39af476345da43fd8f67b1"} Oct 30 19:54:38.680102 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.680011 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.718422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.718315 4959 generic.go:356] "Generic (PLEG): container finished" podID="09dea38e-894f-4826-9d13-2ed301c8d50f" containerID="7e15aecc338cff099b4b9cd46dc8661a6763bf8d96e2474e42f708d7356e33e6" exitCode=0 Oct 30 19:54:38.718422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.718368 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" event={"ID":"09dea38e-894f-4826-9d13-2ed301c8d50f","Type":"ContainerDied","Data":"7e15aecc338cff099b4b9cd46dc8661a6763bf8d96e2474e42f708d7356e33e6"} Oct 30 19:54:38.723090 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.723054 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:54:38.983141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.898646 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\") pod \"09dea38e-894f-4826-9d13-2ed301c8d50f\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " Oct 30 19:54:38.983141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.899014 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\" (UniqueName: \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\") pod \"09dea38e-894f-4826-9d13-2ed301c8d50f\" (UID: \"09dea38e-894f-4826-9d13-2ed301c8d50f\") " Oct 30 19:54:38.983141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.924333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.983141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.924842 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume" pod "09dea38e-894f-4826-9d13-2ed301c8d50f" (UID: "09dea38e-894f-4826-9d13-2ed301c8d50f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:38.983141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:38.963751 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7" pod "09dea38e-894f-4826-9d13-2ed301c8d50f" (UID: "09dea38e-894f-4826-9d13-2ed301c8d50f"). InnerVolumeSpecName "kube-api-access-kxwl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:39.014079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.009128 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09dea38e-894f-4826-9d13-2ed301c8d50f-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.014079 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.009157 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwl7\" (UniqueName: \"kubernetes.io/projected/09dea38e-894f-4826-9d13-2ed301c8d50f-kube-api-access-kxwl7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.096446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.056321 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.096446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.096003 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:54:39.123401 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.123370 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\") pod \"d042ff70-6985-4cd7-a708-511103684116\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " Oct 30 19:54:39.123744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.123713 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\" (UniqueName: \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\") pod \"d042ff70-6985-4cd7-a708-511103684116\" (UID: \"d042ff70-6985-4cd7-a708-511103684116\") " Oct 30 19:54:39.126958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.126925 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume" pod "d042ff70-6985-4cd7-a708-511103684116" (UID: "d042ff70-6985-4cd7-a708-511103684116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:39.145516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.142261 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:54:39.145516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.145458 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5" pod "d042ff70-6985-4cd7-a708-511103684116" (UID: "d042ff70-6985-4cd7-a708-511103684116"). InnerVolumeSpecName "kube-api-access-m6lf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:39.177525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.174012 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.225507 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.225455 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\") pod \"754e1383-7314-4d17-bf79-9c8c46661db8\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " Oct 30 19:54:39.229843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.228929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\") pod \"754e1383-7314-4d17-bf79-9c8c46661db8\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " Oct 30 19:54:39.229843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.228992 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") pod \"754e1383-7314-4d17-bf79-9c8c46661db8\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " Oct 30 19:54:39.229843 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.229044 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\" (UniqueName: \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\") pod \"754e1383-7314-4d17-bf79-9c8c46661db8\" (UID: \"754e1383-7314-4d17-bf79-9c8c46661db8\") " Oct 30 19:54:39.231124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.230978 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca" pod "754e1383-7314-4d17-bf79-9c8c46661db8" (UID: "754e1383-7314-4d17-bf79-9c8c46661db8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:39.234629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.231811 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754e1383-7314-4d17-bf79-9c8c46661db8-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.234629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.231840 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d042ff70-6985-4cd7-a708-511103684116-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.234629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.231855 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lf5\" (UniqueName: \"kubernetes.io/projected/d042ff70-6985-4cd7-a708-511103684116-kube-api-access-m6lf5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.249046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.249018 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z" pod "754e1383-7314-4d17-bf79-9c8c46661db8" (UID: "754e1383-7314-4d17-bf79-9c8c46661db8"). InnerVolumeSpecName "kube-api-access-8b55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:39.250659 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.250631 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls" pod "754e1383-7314-4d17-bf79-9c8c46661db8" (UID: "754e1383-7314-4d17-bf79-9c8c46661db8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:39.251229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.251200 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs" pod "754e1383-7314-4d17-bf79-9c8c46661db8" (UID: "754e1383-7314-4d17-bf79-9c8c46661db8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:39.333170 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.333134 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.333170 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.333170 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754e1383-7314-4d17-bf79-9c8c46661db8-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.333170 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.333180 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b55z\" (UniqueName: \"kubernetes.io/projected/754e1383-7314-4d17-bf79-9c8c46661db8-kube-api-access-8b55z\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:54:39.745814 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.745776 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" event={"ID":"d042ff70-6985-4cd7-a708-511103684116","Type":"ContainerDied","Data":"1a1ed53a6487e7d7a63fa9925b002f48b8cad70cbd53ecc94796a0c266aba529"} Oct 30 19:54:39.746430 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.746409 4959 scope.go:122] "RemoveContainer" containerID="c08d8462e5ba1a0fb4edbc4efcac18cb8798e4eded39af476345da43fd8f67b1" Oct 30 19:54:39.746727 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.746711 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4" Oct 30 19:54:39.750139 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.750107 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" event={"ID":"09dea38e-894f-4826-9d13-2ed301c8d50f","Type":"ContainerDied","Data":"fd6a56115b9f4771a865b15463a5ce9bdc50e39c3507a3443950986a739fbba0"} Oct 30 19:54:39.750244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.750227 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968" Oct 30 19:54:39.762585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.762512 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" event={"ID":"754e1383-7314-4d17-bf79-9c8c46661db8","Type":"ContainerDied","Data":"6b6c1c5e571b786b335b26390d40c1bc2a17eebd0e355590c425b3edbc739d13"} Oct 30 19:54:39.762585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.762567 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5" Oct 30 19:54:39.764236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.764219 4959 scope.go:122] "RemoveContainer" containerID="7e15aecc338cff099b4b9cd46dc8661a6763bf8d96e2474e42f708d7356e33e6" Oct 30 19:54:39.778172 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.778150 4959 scope.go:122] "RemoveContainer" containerID="d11c155e1a9b8614ff9542aa4c943d2a909730671c83c44cf25cd492cd22ec38" Oct 30 19:54:39.840750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.840656 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:54:39.857358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.857329 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c9c6d81d-65486c5676-ct6s4"] Oct 30 19:54:39.884320 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.884291 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:54:39.888488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.888458 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c9c6d81d-6cfd4675bd-j4sx5"] Oct 30 19:54:39.925116 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.925085 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:54:39.931286 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:39.931261 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c9c6d81d-8b46f798c-vb968"] Oct 30 19:54:40.001010 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:40.000926 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dea38e-894f-4826-9d13-2ed301c8d50f" path="/var/lib/kubelet/pods/09dea38e-894f-4826-9d13-2ed301c8d50f/volumes" Oct 30 19:54:40.001629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:40.001612 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754e1383-7314-4d17-bf79-9c8c46661db8" path="/var/lib/kubelet/pods/754e1383-7314-4d17-bf79-9c8c46661db8/volumes" Oct 30 19:54:40.002036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:40.002022 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d042ff70-6985-4cd7-a708-511103684116" path="/var/lib/kubelet/pods/d042ff70-6985-4cd7-a708-511103684116/volumes" Oct 30 19:54:41.041540 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:41.036844 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.566523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:42.565896 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:43.351763 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:43.351736 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:54:43.351918 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:43.351809 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:54:44.091664 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:44.091631 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.657720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.657688 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:54:45.658551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658528 4959 cpu_manager.go:414] "containerMap: removing container" podUID="09dea38e-894f-4826-9d13-2ed301c8d50f" containerName="payload-pre-processing" Oct 30 19:54:45.658551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658550 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dea38e-894f-4826-9d13-2ed301c8d50f" containerName="payload-pre-processing" Oct 30 19:54:45.658732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658621 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d042ff70-6985-4cd7-a708-511103684116" containerName="payload-processing" Oct 30 19:54:45.658732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658630 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d042ff70-6985-4cd7-a708-511103684116" containerName="payload-processing" Oct 30 19:54:45.658732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658641 4959 cpu_manager.go:414] "containerMap: removing container" podUID="754e1383-7314-4d17-bf79-9c8c46661db8" containerName="maas-api" Oct 30 19:54:45.658732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658648 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e1383-7314-4d17-bf79-9c8c46661db8" containerName="maas-api" Oct 30 19:54:45.658926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658804 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dea38e-894f-4826-9d13-2ed301c8d50f" containerName="payload-pre-processing" Oct 30 19:54:45.658926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658829 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d042ff70-6985-4cd7-a708-511103684116" containerName="payload-processing" Oct 30 19:54:45.658926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.658839 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e1383-7314-4d17-bf79-9c8c46661db8" containerName="maas-api" Oct 30 19:54:45.664022 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.663999 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.684271 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.684246 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:54:45.689424 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.689400 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c53b7887-serving-cert\"" Oct 30 19:54:45.742331 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.742292 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.746922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.746897 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.747105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.747089 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.747253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.747236 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.747329 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.747299 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfcc\" (UniqueName: \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.849236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.849205 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.849408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.849248 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfcc\" (UniqueName: \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.849408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.849316 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.849586 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.849446 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.850071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.850046 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.851910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.851888 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.852006 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.851953 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.859390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.859361 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.870255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.870230 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfcc\" (UniqueName: \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\") pod \"maas-api-e2e-mt-c53b7887-75f584459c-qcjbx\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:45.983033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.982727 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:46.008552 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:45.998830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.244589 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:54:46.242926 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:54:46.649849 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:46.649435 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:54:46.652991 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:54:46.652963 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a75cc7_2da6_4cdb_a30c_fa21634df5f0.slice/crio-5f1c7a8236d6d4aebd5724bf18580acc88c434ec2ac1a36f1ec5a275b7cf0e0e WatchSource:0}: Error finding container 5f1c7a8236d6d4aebd5724bf18580acc88c434ec2ac1a36f1ec5a275b7cf0e0e: Status 404 returned error can't find the container with id 5f1c7a8236d6d4aebd5724bf18580acc88c434ec2ac1a36f1ec5a275b7cf0e0e Oct 30 19:54:46.815667 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:46.815632 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" event={"ID":"73a75cc7-2da6-4cdb-a30c-fa21634df5f0","Type":"ContainerStarted","Data":"5f1c7a8236d6d4aebd5724bf18580acc88c434ec2ac1a36f1ec5a275b7cf0e0e"} Oct 30 19:54:47.595219 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:47.595182 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.838521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:47.836539 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" event={"ID":"73a75cc7-2da6-4cdb-a30c-fa21634df5f0","Type":"ContainerStarted","Data":"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4"} Oct 30 19:54:47.839105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:47.839081 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:47.885426 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:47.885348 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" podStartSLOduration=2.527657514 podStartE2EDuration="2.885327509s" totalImagesPullingTime="357.669995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:47.881067367 +0000 UTC m=+1212.010301287" watchObservedRunningTime="2026-10-30 19:54:47.885327509 +0000 UTC m=+1212.014561431" Oct 30 19:54:48.094567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:48.085361 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:50.145658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:50.145622 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-gqnwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:54:50.146076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:50.145669 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.200555 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:50.200051 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.152458 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:52.152420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:53.920057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:53.920024 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.170624 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:54.170540 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.859779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:54.859752 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:54:55.483635 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:55.483587 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.615448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:56.615410 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.731456 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:56.731417 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:57.091618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:57.091577 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:58.110121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:58.110085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:58.224272 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:58.224244 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:59.000263 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:54:59.000129 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.463602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:00.463569 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:55:00.464424 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:00.464389 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" podUID="3916ece0-cabe-44a6-9511-f538c677a751" containerName="maas-api" containerID="cri-o://482c650391c19a19c02b2aead89a610e104b7ebedcfa797e5693846fac679fde" gracePeriod=30 Oct 30 19:55:00.528884 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:00.528850 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.974950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:00.965147 4959 generic.go:356] "Generic (PLEG): container finished" podID="3916ece0-cabe-44a6-9511-f538c677a751" containerID="482c650391c19a19c02b2aead89a610e104b7ebedcfa797e5693846fac679fde" exitCode=0 Oct 30 19:55:00.974950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:00.965220 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" event={"ID":"3916ece0-cabe-44a6-9511-f538c677a751","Type":"ContainerDied","Data":"482c650391c19a19c02b2aead89a610e104b7ebedcfa797e5693846fac679fde"} Oct 30 19:55:01.002729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.002703 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:55:01.106721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.106689 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\" (UniqueName: \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\") pod \"3916ece0-cabe-44a6-9511-f538c677a751\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " Oct 30 19:55:01.106920 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.106764 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\") pod \"3916ece0-cabe-44a6-9511-f538c677a751\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " Oct 30 19:55:01.106990 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.106954 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\") pod \"3916ece0-cabe-44a6-9511-f538c677a751\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " Oct 30 19:55:01.107105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.107091 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\") pod \"3916ece0-cabe-44a6-9511-f538c677a751\" (UID: \"3916ece0-cabe-44a6-9511-f538c677a751\") " Oct 30 19:55:01.107340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.107312 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca" pod "3916ece0-cabe-44a6-9511-f538c677a751" (UID: "3916ece0-cabe-44a6-9511-f538c677a751"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:01.107869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.107840 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3916ece0-cabe-44a6-9511-f538c677a751-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:55:01.109329 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.109297 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls" pod "3916ece0-cabe-44a6-9511-f538c677a751" (UID: "3916ece0-cabe-44a6-9511-f538c677a751"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:01.109700 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.109671 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs" pod "3916ece0-cabe-44a6-9511-f538c677a751" (UID: "3916ece0-cabe-44a6-9511-f538c677a751"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:01.109789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.109677 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs" pod "3916ece0-cabe-44a6-9511-f538c677a751" (UID: "3916ece0-cabe-44a6-9511-f538c677a751"). InnerVolumeSpecName "kube-api-access-47mxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:01.139685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.139655 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.209107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.209071 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:55:01.209107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.209101 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mxs\" (UniqueName: \"kubernetes.io/projected/3916ece0-cabe-44a6-9511-f538c677a751-kube-api-access-47mxs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:55:01.209107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.209112 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3916ece0-cabe-44a6-9511-f538c677a751-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:55:01.314308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.314225 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.492621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.492567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.618680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.618298 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.975237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.975200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" event={"ID":"3916ece0-cabe-44a6-9511-f538c677a751","Type":"ContainerDied","Data":"bef599b143aa7df92b783a50398d431f8b969adf690d09a3ec7914ec72d63a34"} Oct 30 19:55:01.975463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.975254 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw" Oct 30 19:55:01.975463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:01.975274 4959 scope.go:122] "RemoveContainer" containerID="482c650391c19a19c02b2aead89a610e104b7ebedcfa797e5693846fac679fde" Oct 30 19:55:02.034823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.034791 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:55:02.049942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.049914 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-75082b11-7f9c875b88-lrspw"] Oct 30 19:55:02.365453 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.365374 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.784909 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.784875 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 19:55:02.785626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.785606 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3916ece0-cabe-44a6-9511-f538c677a751" containerName="maas-api" Oct 30 19:55:02.785626 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.785624 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3916ece0-cabe-44a6-9511-f538c677a751" containerName="maas-api" Oct 30 19:55:02.785820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.785728 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3916ece0-cabe-44a6-9511-f538c677a751" containerName="maas-api" Oct 30 19:55:02.791488 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.791456 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.821206 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.821180 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-043892-openshift-default-dockercfg-hvz52\"" Oct 30 19:55:02.840473 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.834797 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 19:55:02.840473 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.839354 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.849647 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849618 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.849801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849697 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.849801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849737 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfj7\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.849922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849829 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.849922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849887 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.849922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.849904 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.850087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.850033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.850087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.850052 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.850087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.850077 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.947706 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.947661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.951408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951383 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951445 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfj7\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951506 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951567 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951594 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951795 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951777 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951868 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951811 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951868 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951811 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951888 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.951980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.951932 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.952088 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.952030 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.952247 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.952225 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.952481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.952409 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.952561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.952488 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.954043 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.954023 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:02.954581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:02.954563 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:03.050572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.034553 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfj7\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:03.101518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.098964 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\") pod \"e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:03.129521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.126409 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:03.655600 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:55:03.655571 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1d9453_4769_4ab6_8b80_b767c3ba2bd8.slice/crio-ba0826435f82d4585f5661e8970e40b5642e1a1a5804c05b204f0811e701a642 WatchSource:0}: Error finding container ba0826435f82d4585f5661e8970e40b5642e1a1a5804c05b204f0811e701a642: Status 404 returned error can't find the container with id ba0826435f82d4585f5661e8970e40b5642e1a1a5804c05b204f0811e701a642 Oct 30 19:55:03.656980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.656956 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 19:55:03.658169 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.658138 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:03.658246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.658202 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:03.658246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.658236 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:03.823544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:03.823477 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:04.040314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.040244 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3916ece0-cabe-44a6-9511-f538c677a751" path="/var/lib/kubelet/pods/3916ece0-cabe-44a6-9511-f538c677a751/volumes" Oct 30 19:55:04.089145 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.089104 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerStarted","Data":"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d"} Oct 30 19:55:04.089328 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.089165 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerStarted","Data":"ba0826435f82d4585f5661e8970e40b5642e1a1a5804c05b204f0811e701a642"} Oct 30 19:55:04.127430 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.127388 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:04.128686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.128663 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:55:04.128804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.128708 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:55:04.141091 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:04.141049 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podStartSLOduration=2.141036993 podStartE2EDuration="2.141036993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:04.139468202 +0000 UTC m=+1228.268702145" watchObservedRunningTime="2026-10-30 19:55:04.141036993 +0000 UTC m=+1228.270270911" Oct 30 19:55:05.131307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:05.131283 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:06.010691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:06.010662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:06.152148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:06.152120 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:06.164655 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:06.164625 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:55:07.059783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:07.059745 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.166695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:07.166666 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:08.021461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:08.021425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:08.937737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:08.937696 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.058422 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.058392 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 19:55:09.064315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.064284 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.067267 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.067241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.083068 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.083049 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 19:55:09.087642 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.087621 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.096129 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.096100 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 19:55:09.114021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.109087 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-043892\"" Oct 30 19:55:09.114021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.109240 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-043892-dockercfg-66xcm\"" Oct 30 19:55:09.117919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.117610 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 19:55:09.178019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.177986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.223146 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.223068 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 19:55:09.227889 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.227855 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241091 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241172 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241212 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241282 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241337 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241404 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241558 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241707 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241741 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6pw\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241782 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.243562 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.241889 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmsd\" (UniqueName: \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.252052 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.251942 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 19:55:09.376520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.347406 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.389617 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6pw\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.389687 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.389799 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmsd\" (UniqueName: \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.389954 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390003 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390030 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390064 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390136 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390191 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390239 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288nk\" (UniqueName: \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390292 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.390455 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.358026 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.391455 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.392448 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.392725 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.393517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.392947 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.410525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.406787 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.456521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.454345 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.471397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.471033 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.483625 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.474993 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmsd\" (UniqueName: \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\") pod \"payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.496606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.492722 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6pw\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.496606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.493982 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.496606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.494099 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288nk\" (UniqueName: \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.532585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.530058 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-42srx\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.532585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.530257 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.533840 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.533812 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288nk\" (UniqueName: \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\") pod \"payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.553046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.552769 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:09.684246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.683820 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:09.720364 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.720043 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:09.845741 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:09.845483 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.159555 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.152935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.277618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.277579 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.365278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.365247 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 19:55:10.368399 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.368357 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:10.369483 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.368444 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:10.369483 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.368487 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:55:10.402264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.402207 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 19:55:10.405126 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:55:10.405096 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7ecaa8_b517_4855_83ba_8e41ee262562.slice/crio-e5d542d2fbc5c53d4aa0f7bb8d96cda521a3d7e0ab3b47b381f4ca3c1f6427e6 WatchSource:0}: Error finding container e5d542d2fbc5c53d4aa0f7bb8d96cda521a3d7e0ab3b47b381f4ca3c1f6427e6: Status 404 returned error can't find the container with id e5d542d2fbc5c53d4aa0f7bb8d96cda521a3d7e0ab3b47b381f4ca3c1f6427e6 Oct 30 19:55:10.405402 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.405381 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.480060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.479998 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 19:55:10.484032 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:55:10.482234 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f83ae7_8073_4199_be91_5ff3531198d9.slice/crio-2422f3312ea0c92ae22042da95a18cc9b7d0e7045f96290ef99ab1e4a912d951 WatchSource:0}: Error finding container 2422f3312ea0c92ae22042da95a18cc9b7d0e7045f96290ef99ab1e4a912d951: Status 404 returned error can't find the container with id 2422f3312ea0c92ae22042da95a18cc9b7d0e7045f96290ef99ab1e4a912d951 Oct 30 19:55:10.621519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:10.618686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:11.242248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.242219 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" event={"ID":"d0c07e0b-d292-4832-81c2-d5198affed5b","Type":"ContainerStarted","Data":"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c"} Oct 30 19:55:11.242756 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.242600 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" event={"ID":"d0c07e0b-d292-4832-81c2-d5198affed5b","Type":"ContainerStarted","Data":"cbdbf3c9bc3700f0e409bfa43974c8b91f5625799c7bd08fcc175a903be733ab"} Oct 30 19:55:11.245376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.245352 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" event={"ID":"7f7ecaa8-b517-4855-83ba-8e41ee262562","Type":"ContainerStarted","Data":"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e"} Oct 30 19:55:11.245573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.245557 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" event={"ID":"7f7ecaa8-b517-4855-83ba-8e41ee262562","Type":"ContainerStarted","Data":"e5d542d2fbc5c53d4aa0f7bb8d96cda521a3d7e0ab3b47b381f4ca3c1f6427e6"} Oct 30 19:55:11.245691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.245681 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:11.247195 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.247172 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" event={"ID":"c8f83ae7-8073-4199-be91-5ff3531198d9","Type":"ContainerStarted","Data":"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5"} Oct 30 19:55:11.247332 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.247218 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" event={"ID":"c8f83ae7-8073-4199-be91-5ff3531198d9","Type":"ContainerStarted","Data":"2422f3312ea0c92ae22042da95a18cc9b7d0e7045f96290ef99ab1e4a912d951"} Oct 30 19:55:11.422115 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.422049 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podStartSLOduration=2.4220287 podStartE2EDuration="2.4220287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.342857216 +0000 UTC m=+1235.472091138" watchObservedRunningTime="2026-10-30 19:55:11.4220287 +0000 UTC m=+1235.551262623" Oct 30 19:55:11.509580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.506678 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" podStartSLOduration=2.506653291 podStartE2EDuration="2.506653291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.473838692 +0000 UTC m=+1235.603072614" watchObservedRunningTime="2026-10-30 19:55:11.506653291 +0000 UTC m=+1235.635887213" Oct 30 19:55:11.509580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.507048 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" podStartSLOduration=2.507036666 podStartE2EDuration="2.507036666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.4245293 +0000 UTC m=+1235.553763217" watchObservedRunningTime="2026-10-30 19:55:11.507036666 +0000 UTC m=+1235.636270588" Oct 30 19:55:11.689550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.689523 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:11.689923 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.689899 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:11.690088 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.690064 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:11.810577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:11.809835 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:12.323309 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:12.323272 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:12.739161 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:12.735683 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:12.739161 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:12.738560 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:13.648452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:13.648375 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.687154 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:13.687113 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.687335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:13.687188 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:14.533367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:14.533333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:14.685682 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:14.685648 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:14.686081 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:14.685709 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:15.072641 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:15.071859 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:15.685805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:15.685770 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:15.686191 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:15.685823 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:16.094594 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:55:16.094461 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:55:16.686011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:16.685973 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:16.686487 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:16.686030 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:16.823246 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:16.823178 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:17.036023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:17.035943 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:17.409888 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:17.409858 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:17.685723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:17.685643 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-42srx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:55:17.685723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:17.685688 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 19:55:18.490187 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:18.490158 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.521750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:19.521723 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:19.522313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:19.522271 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:19.523133 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:19.523105 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 19:55:20.162070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:20.162039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:20.978943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:20.978912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:21.621159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.621128 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 19:55:21.626950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.626928 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.657602 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.657571 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 19:55:21.693821 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.693784 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.694007 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.693837 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.694057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.694011 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.694243 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.694222 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7sg8\" (UniqueName: \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.705230 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.705205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:21.741743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.741705 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-043892-serving-cert\"" Oct 30 19:55:21.801333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.801269 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7sg8\" (UniqueName: \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.834362 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.834262 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:21.842901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.842869 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.867514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.863856 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.867514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.864025 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.869372 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:55:21.867916 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-serving-cert: secret "maas-api-e2e-shared-a-w4-043892-serving-cert" not found Oct 30 19:55:21.869372 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:55:21.868067 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls podName:55503847-60a8-4000-960c-8ea1e58fff3d nodeName:}" failed. No retries permitted until 2026-10-30 19:55:22.367989085 +0000 UTC m=+1246.497222984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls") pod "maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" (UID: "55503847-60a8-4000-960c-8ea1e58fff3d") : secret "maas-api-e2e-shared-a-w4-043892-serving-cert" not found Oct 30 19:55:21.869820 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.869792 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.870598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.870533 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7sg8\" (UniqueName: \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:21.877618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:21.870886 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:22.089910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.089851 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:22.341860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.341833 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 19:55:22.453569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.425350 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:22.481203 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.481167 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:22.541691 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.541658 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:22.915905 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:22.915872 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 19:55:23.334049 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.334022 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 19:55:23.395994 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.395954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:23.530701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.528851 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" event={"ID":"55503847-60a8-4000-960c-8ea1e58fff3d","Type":"ContainerStarted","Data":"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0"} Oct 30 19:55:23.530701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.528924 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" event={"ID":"55503847-60a8-4000-960c-8ea1e58fff3d","Type":"ContainerStarted","Data":"6a5406657706aaba83a7fdec89806190e4cb024034a172f89e28637bb1e09599"} Oct 30 19:55:23.530701 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.530537 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:23.588985 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:23.588914 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" podStartSLOduration=2.299122891 podStartE2EDuration="2.588894797s" totalImagesPullingTime="289.771906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:23.586061288 +0000 UTC m=+1247.715295211" watchObservedRunningTime="2026-10-30 19:55:23.588894797 +0000 UTC m=+1247.718128718" Oct 30 19:55:24.896121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:24.896094 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:26.496831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:26.496795 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:27.984051 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:27.984019 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.160520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:29.157113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.687548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:29.686327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:30.197944 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:30.197910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:30.310517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:30.310288 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:30.566575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:30.563653 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 19:55:31.621992 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:31.621960 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.691922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:32.691893 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:33.197450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:33.197415 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:33.372543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:33.358817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:34.002447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:34.002415 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.328901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:35.328866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.126059 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:37.126018 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.417064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:37.417033 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.536928 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:37.536901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.867076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:38.867041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.079685 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.079575 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.185428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.185393 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.280091 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.280059 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 19:55:39.285969 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.285945 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.298664 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.298629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.302477 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.302448 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 19:55:39.310447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.310412 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c8fa69\"" Oct 30 19:55:39.311096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.311074 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c8fa69-dockercfg-57xfm\"" Oct 30 19:55:39.418229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.414564 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 19:55:39.429523 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.424356 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6td\" (UniqueName: \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.530461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.454228 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.530461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.454966 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.530461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.487386 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 19:55:39.603528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.559722 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.678684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.677023 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6td\" (UniqueName: \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.678684 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.570605 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.771712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.767230 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6td\" (UniqueName: \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\") pod \"payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.782379 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.782311 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.782780 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.782752 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdkxs\" (UniqueName: \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.884111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.884082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdkxs\" (UniqueName: \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.884461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.884219 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.885256 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.885235 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.901665 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.901644 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:39.906385 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.906019 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdkxs\" (UniqueName: \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\") pod \"payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:39.983551 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:39.983398 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:40.215817 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:55:40.215784 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3b038e_72a5_43d9_985e_9e03ae2d8fe2.slice/crio-218f10e0f47136ec60ce3d0913b4b1c85c2a563edc603fcd8ce96d5fef6da5e5 WatchSource:0}: Error finding container 218f10e0f47136ec60ce3d0913b4b1c85c2a563edc603fcd8ce96d5fef6da5e5: Status 404 returned error can't find the container with id 218f10e0f47136ec60ce3d0913b4b1c85c2a563edc603fcd8ce96d5fef6da5e5 Oct 30 19:55:40.216962 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:40.216941 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 19:55:40.365522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:40.355903 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 19:55:40.406470 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:55:40.400215 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e53623f_9873_45ad_bb50_e7dea4b10f2a.slice/crio-e653cfbd54e586f4613fde2154d0ca752f22081ece487c116841ef45df833758 WatchSource:0}: Error finding container e653cfbd54e586f4613fde2154d0ca752f22081ece487c116841ef45df833758: Status 404 returned error can't find the container with id e653cfbd54e586f4613fde2154d0ca752f22081ece487c116841ef45df833758 Oct 30 19:55:40.708771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:40.708747 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:41.088790 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.088749 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" event={"ID":"3e53623f-9873-45ad-bb50-e7dea4b10f2a","Type":"ContainerStarted","Data":"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a"} Oct 30 19:55:41.089276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.088815 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" event={"ID":"3e53623f-9873-45ad-bb50-e7dea4b10f2a","Type":"ContainerStarted","Data":"e653cfbd54e586f4613fde2154d0ca752f22081ece487c116841ef45df833758"} Oct 30 19:55:41.089276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.088849 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:41.090434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.090406 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" event={"ID":"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2","Type":"ContainerStarted","Data":"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0"} Oct 30 19:55:41.090601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.090449 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" event={"ID":"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2","Type":"ContainerStarted","Data":"218f10e0f47136ec60ce3d0913b4b1c85c2a563edc603fcd8ce96d5fef6da5e5"} Oct 30 19:55:41.090601 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.090478 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:41.134232 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.134187 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" podStartSLOduration=2.134173514 podStartE2EDuration="2.134173514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:41.129987037 +0000 UTC m=+1265.259220960" watchObservedRunningTime="2026-10-30 19:55:41.134173514 +0000 UTC m=+1265.263407452" Oct 30 19:55:41.174825 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:41.174777 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" podStartSLOduration=2.174763423 podStartE2EDuration="2.174763423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:41.171174362 +0000 UTC m=+1265.300408284" watchObservedRunningTime="2026-10-30 19:55:41.174763423 +0000 UTC m=+1265.303997343" Oct 30 19:55:42.026653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:42.026613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.541408 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:42.541364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.718710 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:42.718682 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.979199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:42.979167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:43.087114 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:43.087081 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.298346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:44.298308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.863690 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:45.863651 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:46.081289 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:55:46.081258 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:55:46.551779 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:46.551746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:46.657940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:46.657902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:47.594291 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:47.594254 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.303575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.303476 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.416744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.416692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.507721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.507691 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 19:55:48.513773 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.513754 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.532344 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.532315 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 19:55:48.562770 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.562680 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-159b3a-dockercfg-rzjq6\"" Oct 30 19:55:48.562959 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.562875 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-159b3a\"" Oct 30 19:55:48.681015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.680858 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 19:55:48.748680 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.747390 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:48.793036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.792989 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 19:55:48.799525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.797571 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.799525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.797616 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsnr\" (UniqueName: \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.916359 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.902567 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:48.917102 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.917069 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.919591 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.919563 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsnr\" (UniqueName: \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.921525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.921481 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lngq\" (UniqueName: \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:48.923122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.921115 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:48.974253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:48.974219 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsnr\" (UniqueName: \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\") pod \"payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:49.034064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.034035 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lngq\" (UniqueName: \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:49.034250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.034163 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:49.034942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.034920 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:49.055434 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.055402 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lngq\" (UniqueName: \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\") pod \"payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:49.081017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.080987 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:49.130644 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.130612 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:49.305180 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.305152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:49.449347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.449295 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 19:55:49.498412 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:49.498336 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 19:55:50.269376 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.269331 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" event={"ID":"333d21e4-88f9-4bab-997f-2c82518767a2","Type":"ContainerStarted","Data":"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55"} Oct 30 19:55:50.269818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.269402 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" event={"ID":"333d21e4-88f9-4bab-997f-2c82518767a2","Type":"ContainerStarted","Data":"ed07ff725d01d14e750a29478090a2fa796743ff8d9c9be5430fdcfbbbbe6d22"} Oct 30 19:55:50.269818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.269486 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:55:50.271211 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.271179 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" event={"ID":"43d243ba-2779-4a6c-abd0-498701701489","Type":"ContainerStarted","Data":"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab"} Oct 30 19:55:50.271339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.271221 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" event={"ID":"43d243ba-2779-4a6c-abd0-498701701489","Type":"ContainerStarted","Data":"cd6cf74a96d4e0669d46da199f95ecf29f0dad9e070472ec23ec56c71f0bfb3e"} Oct 30 19:55:50.271339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.271246 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:55:50.314264 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.314220 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" podStartSLOduration=2.314205389 podStartE2EDuration="2.314205389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:50.312401131 +0000 UTC m=+1274.441635053" watchObservedRunningTime="2026-10-30 19:55:50.314205389 +0000 UTC m=+1274.443439310" Oct 30 19:55:50.358743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.358689 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" podStartSLOduration=2.3586746 podStartE2EDuration="2.3586746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:50.354833052 +0000 UTC m=+1274.484066974" watchObservedRunningTime="2026-10-30 19:55:50.3586746 +0000 UTC m=+1274.487908522" Oct 30 19:55:50.997640 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:50.997603 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.158267 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:52.158229 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 19:55:52.158807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:52.158648 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 19:55:52.603851 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:52.603761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.059436 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:54.059405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.174037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:54.174005 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.295535 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:54.293115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:55.800175 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:55.800141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:57.371461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:57.371429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:59.009978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:55:59.009930 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:00.606066 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:00.606028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.109703 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:01.109677 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.231138 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:01.230998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.281950 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:01.281914 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 19:56:01.283697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:01.283681 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 19:56:01.492806 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:01.492763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:02.287715 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:02.287683 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.783989 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.783958 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 19:56:03.788714 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.788695 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:03.813825 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.812740 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-159b3a-serving-cert\"" Oct 30 19:56:03.818804 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.817727 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 19:56:03.877109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.877055 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:03.877307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.877162 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:03.877307 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.877216 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:03.877437 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.877342 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6lc\" (UniqueName: \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:03.929060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:03.929028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.008673 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.008799 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.008856 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.009011 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6lc\" (UniqueName: \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.011020 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:04.012691 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-serving-cert: secret "maas-api-e2e-shared-b-w4-159b3a-serving-cert" not found Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:04.012796 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls podName:13faa547-2544-4c5c-8499-4efe064e86af nodeName:}" failed. No retries permitted until 2026-10-30 19:56:04.512775392 +0000 UTC m=+1288.642009310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls") pod "maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" (UID: "13faa547-2544-4c5c-8499-4efe064e86af") : secret "maas-api-e2e-shared-b-w4-159b3a-serving-cert" not found Oct 30 19:56:04.037616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.026957 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.068600 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.068544 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.134173 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.134141 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6lc\" (UniqueName: \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.226802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.226771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.540136 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.540075 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.569192 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.569127 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:04.709033 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:04.709001 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:05.292099 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:05.291978 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 19:56:05.636792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:05.636756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.676850 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:05.676749 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" event={"ID":"13faa547-2544-4c5c-8499-4efe064e86af","Type":"ContainerStarted","Data":"fc444e8c83e890cd1f5764497e64034d4a8bcb10bd098c203f2efb594fba7f8b"} Oct 30 19:56:06.686199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:06.686159 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" event={"ID":"13faa547-2544-4c5c-8499-4efe064e86af","Type":"ContainerStarted","Data":"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1"} Oct 30 19:56:06.686621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:06.686243 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:06.731447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:06.731387 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" podStartSLOduration=3.305546931 podStartE2EDuration="3.731373965s" totalImagesPullingTime="425.827034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:06.728310544 +0000 UTC m=+1290.857544466" watchObservedRunningTime="2026-10-30 19:56:06.731373965 +0000 UTC m=+1290.860607885" Oct 30 19:56:07.182668 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:07.182629 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:56:07.182865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:07.182703 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:07.183316 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:07.183266 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.807462 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:07.807409 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.917101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:07.917051 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.694059 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:08.693972 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.889071 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:08.889029 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:09.318687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:09.318609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:09.446822 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:09.446780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.518217 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:10.518182 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:11.780818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.780785 4959 generic.go:356] "Generic (PLEG): container finished" podID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerID="17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d" exitCode=0 Oct 30 19:56:11.781461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.780860 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerDied","Data":"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d"} Oct 30 19:56:11.781461 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.781314 4959 scope.go:122] "RemoveContainer" containerID="17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d" Oct 30 19:56:11.782960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.782931 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:11.783064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.783001 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:11.783064 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:11.783033 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:12.112447 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:12.112371 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.700250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:12.700221 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 19:56:12.796021 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:12.795988 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerStarted","Data":"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23"} Oct 30 19:56:12.837827 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:12.837798 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.054041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:13.053948 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.126903 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:13.126877 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:56:13.127073 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:13.127054 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:13.127131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:13.127112 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:13.903465 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:13.903436 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.130915 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.130879 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:14.131190 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.131161 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:14.635037 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.635000 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.774890 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.774854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.814303 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.814268 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 19:56:14.819023 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.819000 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:14.840313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.840283 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 19:56:14.884525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.858017 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2224e496\"" Oct 30 19:56:14.894070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.886619 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2224e496-dockercfg-tzc6r\"" Oct 30 19:56:14.894070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.892385 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2kp\" (UniqueName: \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:14.894070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:14.892635 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.037525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.027823 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2kp\" (UniqueName: \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.064520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.038385 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.140521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.101466 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.190576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.179895 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:15.190576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.179955 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:15.306592 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.306557 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 19:56:15.318695 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.315337 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2kp\" (UniqueName: \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.325854 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.325826 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 19:56:15.326176 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.326157 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.444481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.434170 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.444481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.443887 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kc6\" (UniqueName: \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.444481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.436407 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:15.549086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.546314 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kc6\" (UniqueName: \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.549086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.546522 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.612518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.608026 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.638514 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.627358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kc6\" (UniqueName: \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:15.665556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.663924 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.743894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:15.743807 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:16.232377 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:16.216892 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:16.232377 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:16.216980 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:16.247803 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:16.246702 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:56:16.339034 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:56:16.338999 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f74d5e_eeab_410d_8a8e_0dadf151c738.slice/crio-3f92f089ce73631e9edacb1f27b0905cb6ac9c74f20ea03b4b553c43c67e5f32 WatchSource:0}: Error finding container 3f92f089ce73631e9edacb1f27b0905cb6ac9c74f20ea03b4b553c43c67e5f32: Status 404 returned error can't find the container with id 3f92f089ce73631e9edacb1f27b0905cb6ac9c74f20ea03b4b553c43c67e5f32 Oct 30 19:56:16.340234 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:16.340204 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 19:56:16.580973 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:16.580804 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 19:56:17.127056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.127014 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:17.127380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.127066 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:17.211606 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.211563 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" event={"ID":"07f74d5e-eeab-410d-8a8e-0dadf151c738","Type":"ContainerStarted","Data":"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0"} Oct 30 19:56:17.211823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.211651 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" event={"ID":"07f74d5e-eeab-410d-8a8e-0dadf151c738","Type":"ContainerStarted","Data":"3f92f089ce73631e9edacb1f27b0905cb6ac9c74f20ea03b4b553c43c67e5f32"} Oct 30 19:56:17.211823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.211802 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:17.213082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.213057 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" event={"ID":"96832826-53f3-4b64-b4aa-5eb28e9dc279","Type":"ContainerStarted","Data":"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b"} Oct 30 19:56:17.213188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.213099 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" event={"ID":"96832826-53f3-4b64-b4aa-5eb28e9dc279","Type":"ContainerStarted","Data":"8b48e4f1dfb7753021c41d94aa2e5f45354d8cc213da161ec0da21022fc13638"} Oct 30 19:56:17.213188 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.213140 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:17.275673 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.275611 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" podStartSLOduration=3.275572584 podStartE2EDuration="3.275572584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:17.272236826 +0000 UTC m=+1301.401470749" watchObservedRunningTime="2026-10-30 19:56:17.275572584 +0000 UTC m=+1301.404806506" Oct 30 19:56:17.320337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.320275 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" podStartSLOduration=3.320260148 podStartE2EDuration="3.320260148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:17.315720486 +0000 UTC m=+1301.444954410" watchObservedRunningTime="2026-10-30 19:56:17.320260148 +0000 UTC m=+1301.449494140" Oct 30 19:56:17.408365 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:17.408326 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.001860 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.001828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.139524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.139199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.150546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.143605 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:56:18.150546 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.143666 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 19:56:18.486356 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.486303 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.523887 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.523853 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 19:56:18.529938 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.529913 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.559956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.559924 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5c2e32-openshift-default-dockercfg-6hjx8\"" Oct 30 19:56:18.567568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.567538 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 19:56:18.601883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.601847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642200 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642322 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2n84\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642369 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642408 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642480 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642561 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642602 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642680 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.645520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.642801 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.753998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.753909 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.755579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.755219 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.755917 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.755899 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.756101 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.756084 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2n84\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.756204 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.756192 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.756298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.756286 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.756428 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.756412 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.768833 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.766264 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.769162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.769138 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.769321 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.769306 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.769517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.769480 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.770094 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.770072 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.779518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.770791 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.780481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.780450 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.781423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.780731 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.788934 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.788861 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.809381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.806486 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.809825 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.807896 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.821797 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.821766 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2n84\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\") pod \"e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:18.862535 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:18.856811 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:19.157518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.144543 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:56:19.157518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.146463 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:56:19.182522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.169755 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 19:56:19.412327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.412287 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.561516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.550716 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 19:56:19.563381 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.563347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.610083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.610039 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:19.610311 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.610290 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:19.610471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:19.610452 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:56:20.243083 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.243043 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerStarted","Data":"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f"} Oct 30 19:56:20.243282 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.243102 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerStarted","Data":"61aa7b866af615b7239dca413ab18a276b69f8859e18ce41c74158963d101a2b"} Oct 30 19:56:20.255140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.255113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.857439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.857405 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:20.862583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.862563 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:20.911441 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:20.911396 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podStartSLOduration=2.911383949 podStartE2EDuration="2.911383949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:20.320900033 +0000 UTC m=+1304.450133955" watchObservedRunningTime="2026-10-30 19:56:20.911383949 +0000 UTC m=+1305.040617870" Oct 30 19:56:21.250333 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:21.250294 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:21.251209 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:21.251189 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:56:21.528865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:21.528767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.614317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:22.614284 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.725621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:22.725592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.485336 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:23.477269 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.402676 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.402643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.594612 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.594576 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.604177 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.604152 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 19:56:24.608811 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.608796 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:24.630167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.630130 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 19:56:24.638296 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.638271 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5c2e32\"" Oct 30 19:56:24.647598 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.638971 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5c2e32-dockercfg-j72wc\"" Oct 30 19:56:24.799015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.798928 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:24.799658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.799626 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqp4n\" (UniqueName: \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:24.963751 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.962756 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:24.963751 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.963141 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqp4n\" (UniqueName: \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:24.964662 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:24.964634 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:25.048151 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:25.048121 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqp4n\" (UniqueName: \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\") pod \"payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:25.223174 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:25.223144 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:25.495708 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:25.495636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:25.605515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:25.601877 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 19:56:26.184980 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.184946 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:26.311883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.311373 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" event={"ID":"e5c9b1b7-18fe-4476-b31a-8405cd8491b8","Type":"ContainerStarted","Data":"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461"} Oct 30 19:56:26.311883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.311433 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" event={"ID":"e5c9b1b7-18fe-4476-b31a-8405cd8491b8","Type":"ContainerStarted","Data":"9630f99b8fc4ba335df38a5b0c1d79379a847e6aae69902a6102f458019483ae"} Oct 30 19:56:26.311883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.311518 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:26.347335 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.347298 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:26.357687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:26.356819 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" podStartSLOduration=2.35680047 podStartE2EDuration="2.35680047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:26.355644736 +0000 UTC m=+1310.484878658" watchObservedRunningTime="2026-10-30 19:56:26.35680047 +0000 UTC m=+1310.486034392" Oct 30 19:56:27.525444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:27.525410 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:28.224146 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:28.224120 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 19:56:28.224449 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:28.224433 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 19:56:29.204476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:29.204442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:29.342394 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:29.342345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:30.912380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:30.912323 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.259671 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:32.259645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.918125 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:32.918087 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.577972 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.577941 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 19:56:34.582877 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.582861 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.603678 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.603656 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 19:56:34.611120 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.611094 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2224e496-serving-cert\"" Oct 30 19:56:34.671200 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.671172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.754278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.754212 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.756420 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.754675 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.771117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.771082 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwbv\" (UniqueName: \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.780858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.771480 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.791768 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.782993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.895109 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.895067 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.896080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.896032 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.897338 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.897318 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwbv\" (UniqueName: \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.897539 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.897069 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.898161 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.898138 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.898826 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:34.898805 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-serving-cert: secret "maas-api-e2e-mt-2224e496-serving-cert" not found Oct 30 19:56:34.899081 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:34.899068 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls podName:2cd1615f-32ce-49d5-ae8e-77ceeb1e5984 nodeName:}" failed. No retries permitted until 2026-10-30 19:56:35.399046967 +0000 UTC m=+1319.528280880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls") pod "maas-api-e2e-mt-2224e496-d58664b78-6hbxw" (UID: "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984") : secret "maas-api-e2e-mt-2224e496-serving-cert" not found Oct 30 19:56:34.915171 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.915142 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:34.927356 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:34.927327 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwbv\" (UniqueName: \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:35.306813 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:35.306732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:35.425425 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:35.408109 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:35.454720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:35.454677 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") pod \"maas-api-e2e-mt-2224e496-d58664b78-6hbxw\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:35.497128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:35.497101 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:35.951325 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:35.951296 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 19:56:36.477090 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:36.477058 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" event={"ID":"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984","Type":"ContainerStarted","Data":"d172ff11140a0f267f5da49afbc55e52d079b908b07a5cad59f2b3541e4a6ab3"} Oct 30 19:56:36.477216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:36.477151 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:36.477216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:36.477171 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" event={"ID":"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984","Type":"ContainerStarted","Data":"a8a26ded63003ee13f573a20eb3aa4ded48625989766b52b2a69b657f001f5b5"} Oct 30 19:56:36.532248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:36.526470 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" podStartSLOduration=2.116691892 podStartE2EDuration="2.526451842s" totalImagesPullingTime="409.75995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:36.519647518 +0000 UTC m=+1320.648881440" watchObservedRunningTime="2026-10-30 19:56:36.526451842 +0000 UTC m=+1320.655685767" Oct 30 19:56:36.532248 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:36.529764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.368765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:37.368738 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 19:56:37.500107 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:37.500072 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.677149 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:37.677118 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.096862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:38.096781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:39.717517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:39.717239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:40.548771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:40.548737 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:40.661002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:40.660954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:40.898862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:40.898824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.511403 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:41.511358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.491548 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:42.491509 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 19:56:43.489522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:43.489473 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.557351 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:45.557316 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.721515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:45.712966 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.008453 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:46.008322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.067036 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:46.067000 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:56:46.194126 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:46.193995 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.956687 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:46.956649 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:47.456065 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:47.456038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.165130 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:48.165089 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.304689 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:48.304640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.636455 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:48.636416 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.420270 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:49.420230 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.553721 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:49.553690 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.956330 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:49.956295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.129364 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:50.129330 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.327931 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:50.327841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.440082 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:50.440051 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.605998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:50.605923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.341481 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.341451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.469450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.469421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.508362 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.508331 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:56:51.560778 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.560742 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:56:51.656057 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.656020 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.776466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:51.776000 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:52.994097 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:52.994055 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.589650 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.589607 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.726510 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.726467 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.764738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.764707 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:56:53.769435 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.769414 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:53.789253 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.789229 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:56:53.797519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.797477 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8d97a5\"" Oct 30 19:56:53.797643 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.797621 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8d97a5-dockercfg-tmv6m\"" Oct 30 19:56:53.925193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.913139 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:53.925193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:53.913291 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtbb\" (UniqueName: \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.020531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.019380 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtbb\" (UniqueName: \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.020531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.019791 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.154516 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.144674 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.232622 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.209693 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtbb\" (UniqueName: \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\") pod \"payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.384569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.384540 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:54.698663 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:56:54.698624 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c64900_8ff4_4480_a247_80aeb02e9287.slice/crio-5aa52330c73613071febd2b29c219f2e88ce519ce9f741db2cf50f8a66dc8242 WatchSource:0}: Error finding container 5aa52330c73613071febd2b29c219f2e88ce519ce9f741db2cf50f8a66dc8242: Status 404 returned error can't find the container with id 5aa52330c73613071febd2b29c219f2e88ce519ce9f741db2cf50f8a66dc8242 Oct 30 19:56:54.699405 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.699381 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:56:54.953419 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.953345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.965996 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.965964 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" event={"ID":"38c64900-8ff4-4480-a247-80aeb02e9287","Type":"ContainerStarted","Data":"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f"} Oct 30 19:56:54.966167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.966015 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" event={"ID":"38c64900-8ff4-4480-a247-80aeb02e9287","Type":"ContainerStarted","Data":"5aa52330c73613071febd2b29c219f2e88ce519ce9f741db2cf50f8a66dc8242"} Oct 30 19:56:54.966167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:54.966036 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:56:55.016530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:55.016430 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" podStartSLOduration=2.016413913 podStartE2EDuration="2.016413913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:55.012987658 +0000 UTC m=+1339.142221581" watchObservedRunningTime="2026-10-30 19:56:55.016413913 +0000 UTC m=+1339.145647833" Oct 30 19:56:55.069749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:55.069708 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.189521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:55.181550 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.509075 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.508982 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" podUID="2d151067-78ac-4601-8f06-9cb7c40d94d5" containerName="payload-processing" containerID="cri-o://3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868" gracePeriod=25 Oct 30 19:56:56.561353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.561292 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" podUID="e2eea479-871e-48ab-980b-67bbdc837f71" containerName="payload-pre-processing" containerID="cri-o://a38c80072eb4bb2c0cefa8709fdef0ef43819abee8093c7f61379f52126ed98f" gracePeriod=25 Oct 30 19:56:56.715210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.715175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.868717 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.868648 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.879771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.877952 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 19:56:56.883095 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.883074 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:56.908951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.908925 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 19:56:56.913056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.913037 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-93263c-dockercfg-vt9bn\"" Oct 30 19:56:56.917268 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.917243 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-93263c\"" Oct 30 19:56:56.918287 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.918258 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:56.918657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.918565 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lbl\" (UniqueName: \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:56.929034 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:56.929016 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:56:57.033397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.031950 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lbl\" (UniqueName: \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:57.033397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.032069 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:57.033397 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.033153 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:57.165839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.164188 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\") pod \"2d151067-78ac-4601-8f06-9cb7c40d94d5\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " Oct 30 19:56:57.165839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.164461 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\" (UniqueName: \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\") pod \"2d151067-78ac-4601-8f06-9cb7c40d94d5\" (UID: \"2d151067-78ac-4601-8f06-9cb7c40d94d5\") " Oct 30 19:56:57.210649 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.186730 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk" pod "2d151067-78ac-4601-8f06-9cb7c40d94d5" (UID: "2d151067-78ac-4601-8f06-9cb7c40d94d5"). InnerVolumeSpecName "kube-api-access-rdntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:57.241636 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.211230 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lbl\" (UniqueName: \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\") pod \"payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:57.256783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.242035 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume" pod "2d151067-78ac-4601-8f06-9cb7c40d94d5" (UID: "2d151067-78ac-4601-8f06-9cb7c40d94d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:57.256783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.242769 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:57.281527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.268074 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d151067-78ac-4601-8f06-9cb7c40d94d5-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:56:57.281810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.281784 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdntk\" (UniqueName: \"kubernetes.io/projected/2d151067-78ac-4601-8f06-9cb7c40d94d5-kube-api-access-rdntk\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:56:57.281934 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.273256 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:57.297983 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.297942 4959 generic.go:356] "Generic (PLEG): container finished" podID="e2eea479-871e-48ab-980b-67bbdc837f71" containerID="a38c80072eb4bb2c0cefa8709fdef0ef43819abee8093c7f61379f52126ed98f" exitCode=0 Oct 30 19:56:57.298564 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.298532 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" event={"ID":"e2eea479-871e-48ab-980b-67bbdc837f71","Type":"ContainerDied","Data":"a38c80072eb4bb2c0cefa8709fdef0ef43819abee8093c7f61379f52126ed98f"} Oct 30 19:56:57.298793 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.298771 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" event={"ID":"e2eea479-871e-48ab-980b-67bbdc837f71","Type":"ContainerDied","Data":"c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8"} Oct 30 19:56:57.298925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.298910 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6abbc12de280f8cffbabf66326abe8773069196fdd7862dc49de59f8d4a09b8" Oct 30 19:56:57.301223 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.301199 4959 generic.go:356] "Generic (PLEG): container finished" podID="2d151067-78ac-4601-8f06-9cb7c40d94d5" containerID="3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868" exitCode=0 Oct 30 19:56:57.301445 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.301432 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" Oct 30 19:56:57.303597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.303573 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" event={"ID":"2d151067-78ac-4601-8f06-9cb7c40d94d5","Type":"ContainerDied","Data":"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868"} Oct 30 19:56:57.303713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.303692 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths" event={"ID":"2d151067-78ac-4601-8f06-9cb7c40d94d5","Type":"ContainerDied","Data":"1631d5cfa41c85c96e3fd97b83ab602d22e88db922c76035143396b5b0c213a4"} Oct 30 19:56:57.303792 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.303763 4959 scope.go:122] "RemoveContainer" containerID="3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868" Oct 30 19:56:57.350617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.338586 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 19:56:57.350617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.339939 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2d151067-78ac-4601-8f06-9cb7c40d94d5" containerName="payload-processing" Oct 30 19:56:57.350617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.339961 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d151067-78ac-4601-8f06-9cb7c40d94d5" containerName="payload-processing" Oct 30 19:56:57.350617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.340142 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d151067-78ac-4601-8f06-9cb7c40d94d5" containerName="payload-processing" Oct 30 19:56:57.373520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.352050 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:56:57.387528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.377388 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.396364 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.396166 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\" (UniqueName: \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\") pod \"e2eea479-871e-48ab-980b-67bbdc837f71\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " Oct 30 19:56:57.396984 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.396933 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\") pod \"e2eea479-871e-48ab-980b-67bbdc837f71\" (UID: \"e2eea479-871e-48ab-980b-67bbdc837f71\") " Oct 30 19:56:57.398528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.398483 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume" pod "e2eea479-871e-48ab-980b-67bbdc837f71" (UID: "e2eea479-871e-48ab-980b-67bbdc837f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:57.401940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.401893 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w" pod "e2eea479-871e-48ab-980b-67bbdc837f71" (UID: "e2eea479-871e-48ab-980b-67bbdc837f71"). InnerVolumeSpecName "kube-api-access-vk69w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:57.493290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.493257 4959 scope.go:122] "RemoveContainer" containerID="3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868" Oct 30 19:56:57.506392 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:56:57.504697 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868\": container with ID starting with 3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868 not found: ID does not exist" containerID="3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868" Oct 30 19:56:57.506392 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.504764 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868"} err="failed to get container status \"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868\": rpc error: code = NotFound desc = could not find container \"3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868\": container with ID starting with 3c4817fc09d690a5fb5eefb4d5f27962000ec4226b7a1f03b65cdc2dd8386868 not found: ID does not exist" Oct 30 19:56:57.507044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.506784 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 19:56:57.516350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.508188 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzqg\" (UniqueName: \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.516350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.508254 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.516350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.508430 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk69w\" (UniqueName: \"kubernetes.io/projected/e2eea479-871e-48ab-980b-67bbdc837f71-kube-api-access-vk69w\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:56:57.516350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.508455 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2eea479-871e-48ab-980b-67bbdc837f71-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:56:57.609613 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.609579 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.609881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.609865 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzqg\" (UniqueName: \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.612830 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.611440 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.637862 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.637602 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzqg\" (UniqueName: \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\") pod \"payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.701039 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.700957 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:56:57.707281 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.707253 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c53b7887-dfd595754-d4ths"] Oct 30 19:56:57.821025 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.820998 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:57.876143 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:56:57.876098 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353e7a62_4777_4375_b4d7_a06caf2940e1.slice/crio-a7885afa77863ce118e55912fddbe818f7ecc969f5c37b28336db5bec4721642 WatchSource:0}: Error finding container a7885afa77863ce118e55912fddbe818f7ecc969f5c37b28336db5bec4721642: Status 404 returned error can't find the container with id a7885afa77863ce118e55912fddbe818f7ecc969f5c37b28336db5bec4721642 Oct 30 19:56:57.878550 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:57.878521 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 19:56:58.006485 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.006447 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d151067-78ac-4601-8f06-9cb7c40d94d5" path="/var/lib/kubelet/pods/2d151067-78ac-4601-8f06-9cb7c40d94d5/volumes" Oct 30 19:56:58.137611 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:56:58.137577 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb02c7b_6156_4254_9d1c_778d372ca467.slice/crio-2d2ac8dca3c73852545a95771394b16de997d79820db31aa55a07eaa4fdf6580 WatchSource:0}: Error finding container 2d2ac8dca3c73852545a95771394b16de997d79820db31aa55a07eaa4fdf6580: Status 404 returned error can't find the container with id 2d2ac8dca3c73852545a95771394b16de997d79820db31aa55a07eaa4fdf6580 Oct 30 19:56:58.139214 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.139183 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 19:56:58.310755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.310714 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" event={"ID":"353e7a62-4777-4375-b4d7-a06caf2940e1","Type":"ContainerStarted","Data":"423e989ded65f27e193c1acfa1fb5a43872efb98b4bdbe7d1ff31c0c9f29585f"} Oct 30 19:56:58.310755 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.310762 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" event={"ID":"353e7a62-4777-4375-b4d7-a06caf2940e1","Type":"ContainerStarted","Data":"a7885afa77863ce118e55912fddbe818f7ecc969f5c37b28336db5bec4721642"} Oct 30 19:56:58.311009 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.310969 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:56:58.312471 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.312445 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" event={"ID":"9fb02c7b-6156-4254-9d1c-778d372ca467","Type":"ContainerStarted","Data":"7b737274848ccac997cc298e64884294607e091fd20864163120c4b2db5e75d0"} Oct 30 19:56:58.312639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.312527 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:56:58.312639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.312551 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" event={"ID":"9fb02c7b-6156-4254-9d1c-778d372ca467","Type":"ContainerStarted","Data":"2d2ac8dca3c73852545a95771394b16de997d79820db31aa55a07eaa4fdf6580"} Oct 30 19:56:58.313588 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.313570 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg" Oct 30 19:56:58.357365 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.357307 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" podStartSLOduration=2.357292357 podStartE2EDuration="2.357292357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:58.355106174 +0000 UTC m=+1342.484340096" watchObservedRunningTime="2026-10-30 19:56:58.357292357 +0000 UTC m=+1342.486526277" Oct 30 19:56:58.400460 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.400402 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" podStartSLOduration=2.400387555 podStartE2EDuration="2.400387555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:58.397333233 +0000 UTC m=+1342.526567178" watchObservedRunningTime="2026-10-30 19:56:58.400387555 +0000 UTC m=+1342.529621516" Oct 30 19:56:58.439628 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.439591 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:56:58.443525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:58.443505 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c53b7887-6bdd647987-rwkbg"] Oct 30 19:56:59.094339 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:56:59.094294 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.001732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:00.001697 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2eea479-871e-48ab-980b-67bbdc837f71" path="/var/lib/kubelet/pods/e2eea479-871e-48ab-980b-67bbdc837f71/volumes" Oct 30 19:57:00.734525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:00.717822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.496552 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:02.495919 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.971520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:03.955864 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.547014 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:05.546979 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.976774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:05.976746 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:57:07.184448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:07.184417 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.622085 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:08.599240 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.637413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:08.636682 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:57:08.640887 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:08.640829 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" podUID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" containerName="maas-api" containerID="cri-o://6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4" gracePeriod=30 Oct 30 19:57:08.884568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:08.877735 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.295566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.276594 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.341521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.340631 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 19:57:09.451111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.412666 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.463519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.457788 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 19:57:09.814517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.814074 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:57:09.862565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.862176 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.888522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.873250 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\") pod \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " Oct 30 19:57:09.888522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.873348 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\") pod \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " Oct 30 19:57:09.888522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.873788 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\") pod \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " Oct 30 19:57:09.888522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.873841 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\" (UniqueName: \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\") pod \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\" (UID: \"73a75cc7-2da6-4cdb-a30c-fa21634df5f0\") " Oct 30 19:57:09.947391 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.946990 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca" pod "73a75cc7-2da6-4cdb-a30c-fa21634df5f0" (UID: "73a75cc7-2da6-4cdb-a30c-fa21634df5f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:09.955746 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:09.955648 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls" pod "73a75cc7-2da6-4cdb-a30c-fa21634df5f0" (UID: "73a75cc7-2da6-4cdb-a30c-fa21634df5f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:10.031541 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.031510 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:57:10.031752 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.031546 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:57:10.068963 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.068684 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs" pod "73a75cc7-2da6-4cdb-a30c-fa21634df5f0" (UID: "73a75cc7-2da6-4cdb-a30c-fa21634df5f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:10.074985 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.074935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.076015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.075974 4959 generic.go:356] "Generic (PLEG): container finished" podID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" containerID="6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4" exitCode=0 Oct 30 19:57:10.076337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.076310 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" Oct 30 19:57:10.138534 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.138483 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:57:10.457527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.455664 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc" pod "73a75cc7-2da6-4cdb-a30c-fa21634df5f0" (UID: "73a75cc7-2da6-4cdb-a30c-fa21634df5f0"). InnerVolumeSpecName "kube-api-access-tzfcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:10.475527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.461591 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" event={"ID":"73a75cc7-2da6-4cdb-a30c-fa21634df5f0","Type":"ContainerDied","Data":"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4"} Oct 30 19:57:10.475527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.461656 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx" event={"ID":"73a75cc7-2da6-4cdb-a30c-fa21634df5f0","Type":"ContainerDied","Data":"5f1c7a8236d6d4aebd5724bf18580acc88c434ec2ac1a36f1ec5a275b7cf0e0e"} Oct 30 19:57:10.475527 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.461691 4959 scope.go:122] "RemoveContainer" containerID="6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4" Oct 30 19:57:10.505620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.505600 4959 scope.go:122] "RemoveContainer" containerID="6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4" Oct 30 19:57:10.505949 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:57:10.505925 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4\": container with ID starting with 6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4 not found: ID does not exist" containerID="6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4" Oct 30 19:57:10.506147 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.505958 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4"} err="failed to get container status \"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4\": rpc error: code = NotFound desc = could not find container \"6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4\": container with ID starting with 6321c7581454f12abae958fa2f180157d1487a6022fb816d72f765bd31c096b4 not found: ID does not exist" Oct 30 19:57:10.520720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.520689 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfcc\" (UniqueName: \"kubernetes.io/projected/73a75cc7-2da6-4cdb-a30c-fa21634df5f0-kube-api-access-tzfcc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:57:10.723201 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.723120 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:57:10.736272 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:10.736244 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c53b7887-75f584459c-qcjbx"] Oct 30 19:57:11.543165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.543131 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.717892 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719075 4959 cpu_manager.go:414] "containerMap: removing container" podUID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" containerName="maas-api" Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719098 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" containerName="maas-api" Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719130 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e2eea479-871e-48ab-980b-67bbdc837f71" containerName="payload-pre-processing" Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719139 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2eea479-871e-48ab-980b-67bbdc837f71" containerName="payload-pre-processing" Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719393 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" containerName="maas-api" Oct 30 19:57:11.720518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.719412 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2eea479-871e-48ab-980b-67bbdc837f71" containerName="payload-pre-processing" Oct 30 19:57:11.762000 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.761878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.830226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.828654 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 19:57:11.830226 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.828748 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.881549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.880053 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2224e496\"/\"e2e-lifecycle-model-2224e496-kserve-self-signed-certs\"" Oct 30 19:57:11.881549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.880129 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2224e496\"/\"default-dockercfg-vgqbb\"" Oct 30 19:57:11.881549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.880192 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2224e496\"/\"openshift-service-ca.crt\"" Oct 30 19:57:11.881549 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.880525 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2224e496\"/\"kube-root-ca.crt\"" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.971623 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.971779 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vw5\" (UniqueName: \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.971949 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.972136 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.972180 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:11.981044 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:11.972246 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191267 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vw5\" (UniqueName: \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191404 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191642 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191677 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.198528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.191786 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.214525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.210816 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.214525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.211535 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.218531 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.216099 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.275856 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.243320 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.275856 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.246109 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vw5\" (UniqueName: \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.289651 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.281782 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\") pod \"e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.339671 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.339323 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a75cc7-2da6-4cdb-a30c-fa21634df5f0" path="/var/lib/kubelet/pods/73a75cc7-2da6-4cdb-a30c-fa21634df5f0/volumes" Oct 30 19:57:12.485970 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.485874 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:12.822241 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:12.822213 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d5fc9e_e7f6_49d9_8597_bc361ccdd177.slice/crio-0adb890635d2807885b5260b95d3e9ee3b8700071836f5a13910b23a9265c726 WatchSource:0}: Error finding container 0adb890635d2807885b5260b95d3e9ee3b8700071836f5a13910b23a9265c726: Status 404 returned error can't find the container with id 0adb890635d2807885b5260b95d3e9ee3b8700071836f5a13910b23a9265c726 Oct 30 19:57:12.825124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:12.825090 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 19:57:13.385917 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:13.385880 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" event={"ID":"72d5fc9e-e7f6-49d9-8597-bc361ccdd177","Type":"ContainerStarted","Data":"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86"} Oct 30 19:57:13.386148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:13.385931 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" event={"ID":"72d5fc9e-e7f6-49d9-8597-bc361ccdd177","Type":"ContainerStarted","Data":"0adb890635d2807885b5260b95d3e9ee3b8700071836f5a13910b23a9265c726"} Oct 30 19:57:13.439504 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:13.439431 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" podStartSLOduration=2.439411785 podStartE2EDuration="2.439411785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:13.43893999 +0000 UTC m=+1357.568173909" watchObservedRunningTime="2026-10-30 19:57:13.439411785 +0000 UTC m=+1357.568645707" Oct 30 19:57:13.480250 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:13.480205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.486805 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:13.486732 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:15.705712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:15.705680 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.070305 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:57:16.070224 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:57:16.817783 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:16.817739 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.047123 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.047094 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.210876 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.210669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.403056 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.403023 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 19:57:17.421960 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.420961 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.451515 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.451236 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 19:57:17.454439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.454397 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8d97a5-serving-cert\"" Oct 30 19:57:17.466342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.465669 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.466342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.465760 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.466342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.465879 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.466342 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.466057 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jdj\" (UniqueName: \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.499320 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.498643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.567618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.567584 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.567810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.567702 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.567908 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.567888 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jdj\" (UniqueName: \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.568080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.568061 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.568765 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.568742 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.570634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.570604 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.570760 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.570686 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.592614 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.592580 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jdj\" (UniqueName: \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\") pod \"maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:17.615733 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.615702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.744212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.744098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.753187 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:17.752930 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:18.329857 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:18.329050 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.463464 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:18.463426 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1315b_3516_43ed_aeca_a9f0d7587c0e.slice/crio-6ced13c41f9cad8fc7f8ff16ecc5573740e31ed6322c210a575a870181de7af1 WatchSource:0}: Error finding container 6ced13c41f9cad8fc7f8ff16ecc5573740e31ed6322c210a575a870181de7af1: Status 404 returned error can't find the container with id 6ced13c41f9cad8fc7f8ff16ecc5573740e31ed6322c210a575a870181de7af1 Oct 30 19:57:18.466368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:18.466340 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 19:57:18.503111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:18.503082 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.684520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:18.676974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.978365 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:18.967317 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.194520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:19.192938 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.312567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:19.308589 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.547673 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:19.543677 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" event={"ID":"0fb1315b-3516-43ed-aeca-a9f0d7587c0e","Type":"ContainerStarted","Data":"6ced13c41f9cad8fc7f8ff16ecc5573740e31ed6322c210a575a870181de7af1"} Oct 30 19:57:19.833929 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:19.833839 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.962519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:19.953933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.557159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:20.557122 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" event={"ID":"0fb1315b-3516-43ed-aeca-a9f0d7587c0e","Type":"ContainerStarted","Data":"d3e23bc25378852f81c232333e8a5f6bacad0989fc1e60fe3f37234a9b11f3e0"} Oct 30 19:57:20.557616 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:20.557208 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:20.607983 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:20.607930 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" podStartSLOduration=2.71374491 podStartE2EDuration="3.607915689s" totalImagesPullingTime="894.170779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:20.604331718 +0000 UTC m=+1364.733565639" watchObservedRunningTime="2026-10-30 19:57:20.607915689 +0000 UTC m=+1364.737149609" Oct 30 19:57:20.682158 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:20.682125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.241450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.241399 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.292653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.292614 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:57:22.294620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.294579 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:22.486124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.486091 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:22.499031 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.498948 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:22.735052 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:22.735021 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 19:57:23.736266 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.736190 4959 generic.go:356] "Generic (PLEG): container finished" podID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerID="b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f" exitCode=0 Oct 30 19:57:23.736834 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.736255 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerDied","Data":"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f"} Oct 30 19:57:23.736935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.736914 4959 scope.go:122] "RemoveContainer" containerID="b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f" Oct 30 19:57:23.739150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.739117 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:57:23.739290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.739200 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:57:23.739290 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:23.739251 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:57:24.238298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.238270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.359713 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.359672 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.484759 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.484721 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.604566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.604090 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.800565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.796514 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerStarted","Data":"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d"} Oct 30 19:57:24.858039 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.857957 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:57:24.858410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.858294 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:24.858597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:24.858570 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:25.030741 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:25.030680 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.927518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:25.866162 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:25.927518 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:25.866231 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:26.181605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.172787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.376426 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.376387 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 19:57:26.383087 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.383056 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.383934 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.383907 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 19:57:26.393268 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.393237 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.413910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.413881 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a165e5\"" Oct 30 19:57:26.414038 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.413909 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a165e5-dockercfg-x4lw8\"" Oct 30 19:57:26.495576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.495015 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.495576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.495083 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhrl\" (UniqueName: \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.709629 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.709595 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.777568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.751768 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.777568 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.751856 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhrl\" (UniqueName: \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.820865 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.820829 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 19:57:26.846356 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.845222 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:26.909302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.856737 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 19:57:26.909302 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.864992 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.919528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.915176 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhrl\" (UniqueName: \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\") pod \"payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:26.919528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.915286 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:26.919528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:26.915341 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:27.002858 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.002818 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:27.028818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.015666 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt49m\" (UniqueName: \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.028818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.015976 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.057873 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.046923 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 19:57:27.306060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.255981 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.306925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.306887 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt49m\" (UniqueName: \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.311212 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.260043 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.538036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.492767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.716517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.713812 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt49m\" (UniqueName: \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\") pod \"payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.758831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.757643 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:27.758831 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.758813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.973112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:27.973026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.127519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.127126 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.147351 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.147311 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:28.158747 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.147847 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:28.189975 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.189551 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:28.189975 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.189634 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:28.291279 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.291239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.614093 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.614004 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 19:57:28.710108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.710075 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 19:57:28.783484 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.783263 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 19:57:28.799377 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.724537 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:28.844968 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.844933 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pvj\" (UniqueName: \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:28.848567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.845327 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:28.875902 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.875820 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-27a4d3\"" Oct 30 19:57:28.876298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.876279 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-27a4d3-dockercfg-47zrr\"" Oct 30 19:57:28.878901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.878870 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:28.879012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.878925 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:28.931086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.923722 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 19:57:28.931086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.923777 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 19:57:28.931086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:28.923881 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.004525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.004479 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pvj\" (UniqueName: \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:29.004888 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.004858 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:29.005015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.004930 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qjq\" (UniqueName: \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.005015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.004984 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.006019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.005989 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:29.032283 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.028871 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pvj\" (UniqueName: \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\") pod \"payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:29.108823 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.108786 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qjq\" (UniqueName: \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.109047 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.109031 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.110015 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.109879 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.115144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.115118 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 19:57:29.135384 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.135357 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qjq\" (UniqueName: \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\") pod \"payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.229358 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.229327 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" event={"ID":"ee02872a-a046-4e3e-a6b3-8a31751cff5e","Type":"ContainerStarted","Data":"c9933a269556c3c1b26ce706835c843d18aaf3a21b00974c6f8892f095d15917"} Oct 30 19:57:29.231970 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.231937 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" event={"ID":"f3cb41b8-04f3-4056-86cd-e674bda00fa5","Type":"ContainerStarted","Data":"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e"} Oct 30 19:57:29.232136 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.231998 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" event={"ID":"f3cb41b8-04f3-4056-86cd-e674bda00fa5","Type":"ContainerStarted","Data":"efda0b8bdf75863ae987268f12ddb3fc6351a0b9c4ce9cd9c03a736a3950322c"} Oct 30 19:57:29.232136 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.232050 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:29.234466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.234447 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:29.303775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.300163 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" podStartSLOduration=3.300123493 podStartE2EDuration="3.300123493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:29.287746842 +0000 UTC m=+1373.416980758" watchObservedRunningTime="2026-10-30 19:57:29.300123493 +0000 UTC m=+1373.429357415" Oct 30 19:57:29.303775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.301095 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:29.433208 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.433171 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.830612 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:29.830582 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf2369e_1631_445b_ac9a_1eef4b74d87a.slice/crio-793d9e12c08ffe41e090dd615360a3dc7ffbce1c2eb07cd5c524aca9a446e387 WatchSource:0}: Error finding container 793d9e12c08ffe41e090dd615360a3dc7ffbce1c2eb07cd5c524aca9a446e387: Status 404 returned error can't find the container with id 793d9e12c08ffe41e090dd615360a3dc7ffbce1c2eb07cd5c524aca9a446e387 Oct 30 19:57:29.833369 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.833343 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 19:57:29.869153 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.869113 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:29.870538 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.870482 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:29.884924 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:29.884895 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142619ed_6462_41a5_aa2d_c0faf35bbf48.slice/crio-310a5849141d4b16537aee065d099c15cc796e241fd65e0a002efa0c963b6885 WatchSource:0}: Error finding container 310a5849141d4b16537aee065d099c15cc796e241fd65e0a002efa0c963b6885: Status 404 returned error can't find the container with id 310a5849141d4b16537aee065d099c15cc796e241fd65e0a002efa0c963b6885 Oct 30 19:57:29.885690 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:29.885668 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 19:57:30.241384 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.241274 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" event={"ID":"ee02872a-a046-4e3e-a6b3-8a31751cff5e","Type":"ContainerStarted","Data":"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4"} Oct 30 19:57:30.241384 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.241347 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:30.243002 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.242973 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" event={"ID":"baf2369e-1631-445b-ac9a-1eef4b74d87a","Type":"ContainerStarted","Data":"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880"} Oct 30 19:57:30.243122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.243024 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:30.243122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.243044 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" event={"ID":"baf2369e-1631-445b-ac9a-1eef4b74d87a","Type":"ContainerStarted","Data":"793d9e12c08ffe41e090dd615360a3dc7ffbce1c2eb07cd5c524aca9a446e387"} Oct 30 19:57:30.244719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.244688 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" event={"ID":"142619ed-6462-41a5-aa2d-c0faf35bbf48","Type":"ContainerStarted","Data":"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20"} Oct 30 19:57:30.244836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.244730 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" event={"ID":"142619ed-6462-41a5-aa2d-c0faf35bbf48","Type":"ContainerStarted","Data":"310a5849141d4b16537aee065d099c15cc796e241fd65e0a002efa0c963b6885"} Oct 30 19:57:30.244836 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.244767 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:30.275095 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.275060 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.345810 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.345760 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" podStartSLOduration=4.34574547 podStartE2EDuration="4.34574547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:30.303321616 +0000 UTC m=+1374.432555537" watchObservedRunningTime="2026-10-30 19:57:30.34574547 +0000 UTC m=+1374.474979391" Oct 30 19:57:30.346509 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.346463 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" podStartSLOduration=2.346454715 podStartE2EDuration="2.346454715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:30.343710436 +0000 UTC m=+1374.472944372" watchObservedRunningTime="2026-10-30 19:57:30.346454715 +0000 UTC m=+1374.475688637" Oct 30 19:57:30.397432 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.397381 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" podStartSLOduration=2.397366633 podStartE2EDuration="2.397366633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:30.393942445 +0000 UTC m=+1374.523176364" watchObservedRunningTime="2026-10-30 19:57:30.397366633 +0000 UTC m=+1374.526600554" Oct 30 19:57:30.864585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.858909 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:57:30.864585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:30.858978 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:57:31.865723 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:31.865695 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:57:31.866236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:31.866140 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:57:31.866943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:31.866927 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 19:57:31.997193 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:31.997145 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.068725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:33.068695 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.187978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:33.187952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.978726 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:33.978692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.793999 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.793964 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.912658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.912619 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 19:57:34.915337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.915309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.918861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.918845 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:34.936406 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.936385 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 19:57:34.944571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.944547 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0506c9\"" Oct 30 19:57:34.981997 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.981973 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 19:57:34.986824 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.986802 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:34.987669 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.987640 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdltd\" (UniqueName: \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:34.987982 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:34.987955 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.003674 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.003652 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0506c9-dockercfg-h8jk7\"" Oct 30 19:57:35.009961 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.009941 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 19:57:35.135627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.131107 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.135627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.131220 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdltd\" (UniqueName: \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.135627 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.132695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.249697 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.249649 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.250314 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.250288 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr52g\" (UniqueName: \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.418977 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.406793 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr52g\" (UniqueName: \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.423181 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.422896 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.476566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.472679 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.476566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.473562 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdltd\" (UniqueName: \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\") pod \"payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.496448 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.490487 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr52g\" (UniqueName: \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\") pod \"payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.533806 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.533773 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:35.612947 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.612913 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:35.770259 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.770234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:35.920993 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:35.920953 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 19:57:36.008991 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.008965 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 19:57:36.011044 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:36.011020 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392ba427_9917_49bf_8d60_8459a583f4e8.slice/crio-e4778243a7f8bd8bd6338477304b3a40a07838b8485af459d1f6aebde7b58034 WatchSource:0}: Error finding container e4778243a7f8bd8bd6338477304b3a40a07838b8485af459d1f6aebde7b58034: Status 404 returned error can't find the container with id e4778243a7f8bd8bd6338477304b3a40a07838b8485af459d1f6aebde7b58034 Oct 30 19:57:36.486712 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.486673 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" event={"ID":"ef10935a-addd-444a-b27e-ae3e43183cb2","Type":"ContainerStarted","Data":"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f"} Oct 30 19:57:36.486896 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.486875 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:36.486943 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.486924 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" event={"ID":"ef10935a-addd-444a-b27e-ae3e43183cb2","Type":"ContainerStarted","Data":"2f3080f07360a3b0f19fa3628111737fd70e0307a8de1b47db0d8c8de5f52e74"} Oct 30 19:57:36.488368 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.488346 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" event={"ID":"392ba427-9917-49bf-8d60-8459a583f4e8","Type":"ContainerStarted","Data":"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d"} Oct 30 19:57:36.488463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.488385 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" event={"ID":"392ba427-9917-49bf-8d60-8459a583f4e8","Type":"ContainerStarted","Data":"e4778243a7f8bd8bd6338477304b3a40a07838b8485af459d1f6aebde7b58034"} Oct 30 19:57:36.488463 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.488454 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:36.529544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.529482 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" podStartSLOduration=2.529470622 podStartE2EDuration="2.529470622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:36.527301131 +0000 UTC m=+1380.656535051" watchObservedRunningTime="2026-10-30 19:57:36.529470622 +0000 UTC m=+1380.658704543" Oct 30 19:57:36.575544 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:36.575482 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" podStartSLOduration=2.575470033 podStartE2EDuration="2.575470033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:36.571570663 +0000 UTC m=+1380.700804597" watchObservedRunningTime="2026-10-30 19:57:36.575470033 +0000 UTC m=+1380.704703953" Oct 30 19:57:37.299074 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:37.299040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.956744 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:38.956707 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.136221 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:39.135674 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.248294 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:40.248269 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 19:57:40.787222 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:40.787185 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.278941 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:41.278898 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 19:57:41.279414 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:41.278993 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 19:57:41.279567 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:41.279548 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 19:57:42.270457 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:42.270429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.864278 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:43.864247 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:45.401489 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:45.401459 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.068010 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:57:46.067975 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:57:47.083525 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:47.082191 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.501887 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:47.501847 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 19:57:47.502067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:47.501925 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 19:57:48.681202 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:48.681165 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.500449 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:50.500417 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.128141 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:52.128102 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.490799 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:52.490765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.780581 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:52.780488 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.244542 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:53.244485 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.845446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:54.845401 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.965249 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:54.965221 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:55.121657 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:55.121507 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.757039 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.752602 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 19:57:56.767178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.763458 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 19:57:56.767178 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.763658 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:56.790135 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.790105 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.826676 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.819938 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-27a4d3-serving-cert\"" Oct 30 19:57:56.895521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.894904 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:56.895521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.895106 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8q79\" (UniqueName: \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:56.895521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.895259 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:56.895521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:56.895382 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.000140 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.000097 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.000748 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.000720 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.025905 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.025825 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8q79\" (UniqueName: \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.026803 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.026773 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.027122 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.026196 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.027239 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:57:57.000471 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-serving-cert: secret "maas-api-e2e-shared-b-w6-27a4d3-serving-cert" not found Oct 30 19:57:57.027396 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:57:57.027385 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls podName:9f51f024-9772-4774-ac32-a994be32bc88 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:57.527362425 +0000 UTC m=+1401.656596343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls") pod "maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" (UID: "9f51f024-9772-4774-ac32-a994be32bc88") : secret "maas-api-e2e-shared-b-w6-27a4d3-serving-cert" not found Oct 30 19:57:57.088749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.086091 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.131363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.112648 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8q79\" (UniqueName: \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.138729 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.138662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.335287 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.335202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.587585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.587199 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.620556 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.618141 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:57.703886 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:57.703802 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:57:58.327576 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:58.318049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.379668 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:57:58.379621 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f51f024_9772_4774_ac32_a994be32bc88.slice/crio-d1ac5c430cf0a892d70fc6ae3e80783dff018072a1ba6542bb1947d3ac41fd73 WatchSource:0}: Error finding container d1ac5c430cf0a892d70fc6ae3e80783dff018072a1ba6542bb1947d3ac41fd73: Status 404 returned error can't find the container with id d1ac5c430cf0a892d70fc6ae3e80783dff018072a1ba6542bb1947d3ac41fd73 Oct 30 19:57:58.427914 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:58.427873 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 19:57:58.479553 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:58.479442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.945743 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:58.945684 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" event={"ID":"9f51f024-9772-4774-ac32-a994be32bc88","Type":"ContainerStarted","Data":"d1ac5c430cf0a892d70fc6ae3e80783dff018072a1ba6542bb1947d3ac41fd73"} Oct 30 19:57:59.958715 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:59.958679 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" event={"ID":"9f51f024-9772-4774-ac32-a994be32bc88","Type":"ContainerStarted","Data":"63789771be16c1820433b662009bc835091a40c183d87a387a7efb56a6761701"} Oct 30 19:57:59.959146 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:57:59.958741 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:58:00.022760 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.022704 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" podStartSLOduration=3.529875086 podStartE2EDuration="4.022690174s" totalImagesPullingTime="492.815088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:00.011272165 +0000 UTC m=+1404.140506087" watchObservedRunningTime="2026-10-30 19:58:00.022690174 +0000 UTC m=+1404.151924126" Oct 30 19:58:00.316952 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.316874 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.621255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.621167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.658656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.658624 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 19:58:00.663835 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.663812 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:00.689452 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.689421 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d96913-dockercfg-27jcm\"" Oct 30 19:58:00.690327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.690301 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 19:58:00.697053 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.697025 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d96913\"" Oct 30 19:58:00.765528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.761849 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:00.785558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.779431 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbkn\" (UniqueName: \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:00.981561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.963911 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbkn\" (UniqueName: \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:00.981561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.964049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:00.981561 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:00.965305 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:01.147208 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:01.140653 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbkn\" (UniqueName: \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\") pod \"payload-processing-e2e-api-b-d96913-548c57d588-cbbwm\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:01.278041 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:01.277946 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:01.596716 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:58:01.596692 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473aba10_bccb_47b9_b403_7fb253d16386.slice/crio-605a46cd52be23aaf2fc487690bc43acf2e2173e3435056edd6843fc1646811e WatchSource:0}: Error finding container 605a46cd52be23aaf2fc487690bc43acf2e2173e3435056edd6843fc1646811e: Status 404 returned error can't find the container with id 605a46cd52be23aaf2fc487690bc43acf2e2173e3435056edd6843fc1646811e Oct 30 19:58:01.598925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:01.598851 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 19:58:02.245519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:02.242920 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" event={"ID":"473aba10-bccb-47b9-b403-7fb253d16386","Type":"ContainerStarted","Data":"8583aa8a91bf97bb8b311b2559df04419a61ae4c9ac09d11785eea05ef6a06b9"} Oct 30 19:58:02.245519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:02.243060 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" event={"ID":"473aba10-bccb-47b9-b403-7fb253d16386","Type":"ContainerStarted","Data":"605a46cd52be23aaf2fc487690bc43acf2e2173e3435056edd6843fc1646811e"} Oct 30 19:58:02.251585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:02.249779 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:02.315223 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:02.315171 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" podStartSLOduration=2.315156735 podStartE2EDuration="2.315156735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.30621833 +0000 UTC m=+1406.435452250" watchObservedRunningTime="2026-10-30 19:58:02.315156735 +0000 UTC m=+1406.444390687" Oct 30 19:58:03.517388 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:03.517356 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.598899 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:05.598859 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:06.157578 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:06.157551 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 19:58:06.527998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:06.527911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.174762 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.174730 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.276775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.276745 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 19:58:07.283208 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.283186 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.306486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.306460 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 19:58:07.315435 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.315415 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0506c9-serving-cert\"" Oct 30 19:58:07.327620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.327596 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584vg\" (UniqueName: \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.327774 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.327628 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.327965 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.327942 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.328069 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.328052 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.366287 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.366260 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.429216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.429155 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.429216 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.429209 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.429396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.429305 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584vg\" (UniqueName: \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.429396 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.429324 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.429396 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:07.429375 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-serving-cert: secret "maas-api-e2e-inf-a-0506c9-serving-cert" not found Oct 30 19:58:07.429597 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:07.429452 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls podName:2ca34760-24bb-491a-a5d6-e6ae80e7c5ca nodeName:}" failed. No retries permitted until 2026-10-30 19:58:07.929431735 +0000 UTC m=+1412.058665644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls") pod "maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" (UID: "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca") : secret "maas-api-e2e-inf-a-0506c9-serving-cert" not found Oct 30 19:58:07.429946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.429929 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.431869 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.431852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.476558 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.476278 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-584vg\" (UniqueName: \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:07.547043 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.547008 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.683214 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.683126 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.881589 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.877025 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.990573 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:07.988606 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:08.078128 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:08.074017 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") pod \"maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:08.214519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:08.211862 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:08.401599 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:08.401465 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:08.829319 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:58:08.829281 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca34760_24bb_491a_a5d6_e6ae80e7c5ca.slice/crio-c0c4db68a8670c5b095499a827f8e2d773dd4ca0f1e771d5565753a0e8c4f7a5 WatchSource:0}: Error finding container c0c4db68a8670c5b095499a827f8e2d773dd4ca0f1e771d5565753a0e8c4f7a5: Status 404 returned error can't find the container with id c0c4db68a8670c5b095499a827f8e2d773dd4ca0f1e771d5565753a0e8c4f7a5 Oct 30 19:58:08.877589 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:08.859015 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 19:58:09.444565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:09.441309 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" event={"ID":"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca","Type":"ContainerStarted","Data":"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e"} Oct 30 19:58:09.446630 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:09.446603 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:09.446872 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:09.446640 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" event={"ID":"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca","Type":"ContainerStarted","Data":"c0c4db68a8670c5b095499a827f8e2d773dd4ca0f1e771d5565753a0e8c4f7a5"} Oct 30 19:58:09.490532 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:09.490469 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" podStartSLOduration=2.142520157 podStartE2EDuration="2.490455601s" totalImagesPullingTime="347.935444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:09.487777289 +0000 UTC m=+1413.617011402" watchObservedRunningTime="2026-10-30 19:58:09.490455601 +0000 UTC m=+1413.619689549" Oct 30 19:58:09.718273 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:09.718132 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.591702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:11.591664 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.798526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:11.796123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.286233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:13.286186 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.413347 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:13.413314 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.748566 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:13.743342 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.262718 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:14.262693 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 19:58:15.478296 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:15.478258 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:16.067992 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:16.067957 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:58:16.465690 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:16.465664 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 19:58:17.381698 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:17.381661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.231233 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:19.231197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.878312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:20.878280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.467199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:22.467164 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.923607 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:23.923576 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.736798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.736766 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 19:58:24.742062 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.742045 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.757894 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.757874 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2224e496-openshift-default-dockercfg-xxkmv\"" Oct 30 19:58:24.776433 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.776405 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 19:58:24.850427 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850396 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850511 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nsx\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850621 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850580 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850702 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850682 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850738 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850711 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850775 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.850885 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.850867 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.851028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.851015 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.851111 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.851099 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953112 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953078 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953175 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953228 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953399 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953574 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953515 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953598 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953632 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953655 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nsx\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953694 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.953839 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.953800 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.954090 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.954048 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.954144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.954093 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.954245 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.954221 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.954739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.954712 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.955802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.955760 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.956866 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.956824 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.974464 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.974437 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:24.974609 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:24.974547 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nsx\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:25.058605 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.058537 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:25.383240 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:58:25.383214 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda162056a_76a0_4ea2_9e13_6e7b28623c8b.slice/crio-24996b9a43699dd222524b19c569f4f03fe0b80d47b89f5abaaf22fbc90fab92 WatchSource:0}: Error finding container 24996b9a43699dd222524b19c569f4f03fe0b80d47b89f5abaaf22fbc90fab92: Status 404 returned error can't find the container with id 24996b9a43699dd222524b19c569f4f03fe0b80d47b89f5abaaf22fbc90fab92 Oct 30 19:58:25.384501 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.384467 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 19:58:25.386159 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.386128 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:58:25.386244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.386190 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:58:25.386244 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.386219 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 19:58:25.471124 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.471100 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:25.715143 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.715099 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" event={"ID":"a162056a-76a0-4ea2-9e13-6e7b28623c8b","Type":"ContainerStarted","Data":"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211"} Oct 30 19:58:25.715352 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.715193 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" event={"ID":"a162056a-76a0-4ea2-9e13-6e7b28623c8b","Type":"ContainerStarted","Data":"24996b9a43699dd222524b19c569f4f03fe0b80d47b89f5abaaf22fbc90fab92"} Oct 30 19:58:25.778113 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:25.778060 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podStartSLOduration=1.7780460040000001 podStartE2EDuration="1.778046004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:25.774779085 +0000 UTC m=+1429.904013009" watchObservedRunningTime="2026-10-30 19:58:25.778046004 +0000 UTC m=+1429.907279923" Oct 30 19:58:26.059603 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:26.059535 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:26.060732 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:26.060707 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:26.060861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:26.060751 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:26.989852 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:26.989815 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:27.074524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:27.064912 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:27.074524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:27.064976 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:28.066798 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:28.066766 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:28.067067 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:28.067039 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:28.437749 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:28.437709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.063442 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:29.063386 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:29.064476 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:29.064438 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:29.780528 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:29.780472 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.069927 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:30.069844 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:30.070195 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:30.070164 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:30.849696 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:30.849661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:31.043969 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:31.041239 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:31.064297 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:31.064265 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:31.064571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:31.064547 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:32.075522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:32.071825 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:32.075522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:32.071894 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:33.059389 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:33.059356 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 19:58:33.059590 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:33.059411 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 19:58:33.280802 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:33.280763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.292245 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:34.292169 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:34.293298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:34.293276 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:34.294766 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:34.294739 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 19:58:35.950985 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:35.950935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.082745 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:36.082707 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.433019 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:36.432986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.014840 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:38.014807 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.965110 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:39.965079 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.385317 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:40.385285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.879580 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:40.879449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.999348 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:40.999316 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:41.449072 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:41.449034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:41.723834 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:41.723740 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.063897 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:42.063802 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.411698 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:42.403069 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.612958 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:42.612928 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.530036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:43.530004 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:58:43.530444 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:43.530205 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" containerID="cri-o://27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6" gracePeriod=30 Oct 30 19:58:43.726144 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:43.726113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.940467 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:43.940433 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:44.061265 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:44.061229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:44.553280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:44.553243 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:58:44.553833 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:44.553468 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" containerID="cri-o://04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8" gracePeriod=30 Oct 30 19:58:45.289956 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:45.289925 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.430308 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:45.430272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.582686 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:45.582596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.068375 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:46.068343 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:58:47.416446 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:47.416416 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.543595 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:47.543558 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.678105 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:47.678017 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.966291 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:47.965036 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.152331 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.152295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.828543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.828516 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:58:48.892926 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.892573 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.892964 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893046 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893114 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893215 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893205 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893242 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:48.893363 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893334 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893571 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893549 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893592 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893650 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\") pod \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\" (UID: \"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3\") " Oct 30 19:58:48.893775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893550 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:48.893775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893578 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:48.893775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893729 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:48.894012 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.893989 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:48.894747 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.894720 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.894901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.894753 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.894901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.894767 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.894901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.894780 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.894901 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.894793 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:48.895922 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.895887 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "kube-api-access-kbzw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:48.896345 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.896322 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:48.896610 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.896575 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:48.897467 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:48.897445 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo" pod "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" (UID: "ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:49.013117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.013061 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:49.013350 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.013336 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:49.013450 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.013439 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:49.013565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.013548 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzw8\" (UniqueName: \"kubernetes.io/projected/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3-kube-api-access-kbzw8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:49.551120 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.551084 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.589480 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.589446 4959 generic.go:356] "Generic (PLEG): container finished" podID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerID="27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6" exitCode=0 Oct 30 19:58:49.589672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.589514 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" Oct 30 19:58:49.589672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.589526 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" event={"ID":"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3","Type":"ContainerDied","Data":"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6"} Oct 30 19:58:49.589672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.589600 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln" event={"ID":"ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3","Type":"ContainerDied","Data":"fdab9973bf4213ba58ce64c0743a97880a969b141f149ff22ae8fe9c5615ff0c"} Oct 30 19:58:49.589672 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.589616 4959 scope.go:122] "RemoveContainer" containerID="27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6" Oct 30 19:58:49.625895 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:49.625867 4959 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b71e00_bfeb_48ff_ac4d_7178b78442cf.slice/crio-04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:58:49.668899 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.668876 4959 scope.go:122] "RemoveContainer" containerID="27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6" Oct 30 19:58:49.669212 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:49.669191 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6\": container with ID starting with 27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6 not found: ID does not exist" containerID="27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6" Oct 30 19:58:49.669273 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.669221 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6"} err="failed to get container status \"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6\": rpc error: code = NotFound desc = could not find container \"27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6\": container with ID starting with 27f168022f8cd24fa9bd3e03565b8870b7a9c63b5fbf2834c308c4f2ec8e33b6 not found: ID does not exist" Oct 30 19:58:49.688856 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.688827 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:58:49.690682 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.690655 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.694789 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.694745 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-tvjln"] Oct 30 19:58:49.869303 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.869278 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:58:49.933189 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.933138 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.933543 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.933517 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.940579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.933806 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.940940 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.940909 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.966791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.966755 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.967117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.967092 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.967337 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.967318 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.967683 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.967658 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.967881 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.967860 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\") pod \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\" (UID: \"64b71e00-bfeb-48ff-ac4d-7178b78442cf\") " Oct 30 19:58:49.969036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.945823 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.969167 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.946057 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.981716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.981023 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:49.992613 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:49.990695 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:50.039925 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.038611 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:50.050750 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.050695 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "kube-api-access-dwjds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.077679 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079454 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079484 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079605 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079622 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079651 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079664 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjds\" (UniqueName: \"kubernetes.io/projected/64b71e00-bfeb-48ff-ac4d-7178b78442cf-kube-api-access-dwjds\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.079679 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.088565 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.083909 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" path="/var/lib/kubelet/pods/ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3/volumes" Oct 30 19:58:50.137199 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.136920 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:50.137829 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.137793 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy" pod "64b71e00-bfeb-48ff-ac4d-7178b78442cf" (UID: "64b71e00-bfeb-48ff-ac4d-7178b78442cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:50.209998 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.209963 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.210229 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.210215 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64b71e00-bfeb-48ff-ac4d-7178b78442cf-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:58:50.598653 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.598619 4959 generic.go:356] "Generic (PLEG): container finished" podID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerID="04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8" exitCode=0 Oct 30 19:58:50.598786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.598674 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" Oct 30 19:58:50.598786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.598707 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" event={"ID":"64b71e00-bfeb-48ff-ac4d-7178b78442cf","Type":"ContainerDied","Data":"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8"} Oct 30 19:58:50.598786 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.598764 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq" event={"ID":"64b71e00-bfeb-48ff-ac4d-7178b78442cf","Type":"ContainerDied","Data":"13c0895e3cfd273394f433785320b77cf7132dcdab5ab4a1c0f15a3db1f460e6"} Oct 30 19:58:50.598972 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.598796 4959 scope.go:122] "RemoveContainer" containerID="04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8" Oct 30 19:58:50.611716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.611697 4959 scope.go:122] "RemoveContainer" containerID="04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8" Oct 30 19:58:50.611990 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:50.611974 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8\": container with ID starting with 04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8 not found: ID does not exist" containerID="04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8" Oct 30 19:58:50.612029 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.611998 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8"} err="failed to get container status \"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8\": rpc error: code = NotFound desc = could not find container \"04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8\": container with ID starting with 04ec3dbc3b4b99a1e9d806177604e11290bf7284f24650991f68540e0bb0a9e8 not found: ID does not exist" Oct 30 19:58:50.657423 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.657391 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:58:50.670795 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:50.670766 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-vwslq"] Oct 30 19:58:51.117520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:51.115101 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:52.015237 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.015202 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" path="/var/lib/kubelet/pods/64b71e00-bfeb-48ff-ac4d-7178b78442cf/volumes" Oct 30 19:58:52.931619 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.931591 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 19:58:52.932572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932551 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" Oct 30 19:58:52.932572 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932571 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" Oct 30 19:58:52.932719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932586 4959 cpu_manager.go:414] "containerMap: removing container" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" Oct 30 19:58:52.932719 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932591 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" Oct 30 19:58:52.932791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932742 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0bc6a1-6a54-40d1-b5dd-a8934c7f66c3" containerName="istio-proxy" Oct 30 19:58:52.932791 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.932754 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b71e00-bfeb-48ff-ac4d-7178b78442cf" containerName="istio-proxy" Oct 30 19:58:52.936585 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.936568 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:52.956217 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.956190 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 19:58:52.963801 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:52.963780 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-540a43-serving-cert\"" Oct 30 19:58:53.019631 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.019595 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.021166 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.021140 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.021315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.021195 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.021315 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.021279 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.021479 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.021455 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kws5n\" (UniqueName: \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.123474 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.123437 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.123725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.123510 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.123725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.123546 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.123725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.123618 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kws5n\" (UniqueName: \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.123725 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:53.123675 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-serving-cert: secret "maas-api-e2e-api-b-540a43-serving-cert" not found Oct 30 19:58:53.123725 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.123675 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.124021 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:58:53.123776 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls podName:bde97bd8-1596-4f6d-8091-6fcf87f44da1 nodeName:}" failed. No retries permitted until 2026-10-30 19:58:53.62373933 +0000 UTC m=+1457.752973245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls") pod "maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" (UID: "bde97bd8-1596-4f6d-8091-6fcf87f44da1") : secret "maas-api-e2e-api-b-540a43-serving-cert" not found Oct 30 19:58:53.124254 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.124236 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.126137 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.126119 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.205677 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.200332 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kws5n\" (UniqueName: \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.645148 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.645122 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.645969 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.645936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.647853 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.647833 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") pod \"maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:53.850799 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:53.850775 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:54.160737 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:54.160667 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 19:58:54.161164 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:58:54.161038 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde97bd8_1596_4f6d_8091_6fcf87f44da1.slice/crio-7eea5af8956e899efa11911a62c2c543a704304ad2343d1fde51ec630cddee6d WatchSource:0}: Error finding container 7eea5af8956e899efa11911a62c2c543a704304ad2343d1fde51ec630cddee6d: Status 404 returned error can't find the container with id 7eea5af8956e899efa11911a62c2c543a704304ad2343d1fde51ec630cddee6d Oct 30 19:58:54.162819 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:54.162798 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:58:54.723519 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:54.720836 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" event={"ID":"bde97bd8-1596-4f6d-8091-6fcf87f44da1","Type":"ContainerStarted","Data":"7eea5af8956e899efa11911a62c2c543a704304ad2343d1fde51ec630cddee6d"} Oct 30 19:58:54.965989 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:54.965945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.348530 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.341436 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.729935 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.729898 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" event={"ID":"bde97bd8-1596-4f6d-8091-6fcf87f44da1","Type":"ContainerStarted","Data":"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff"} Oct 30 19:58:55.730134 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.730003 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:58:55.780837 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.780769 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" podStartSLOduration=3.43728025 podStartE2EDuration="3.780747848s" totalImagesPullingTime="343.467598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:55.776349301 +0000 UTC m=+1459.905583229" watchObservedRunningTime="2026-10-30 19:58:55.780747848 +0000 UTC m=+1459.909981771" Oct 30 19:58:55.816121 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.816095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.992666 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:55.992582 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.145521 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:56.134708 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.786522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:56.786147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.390764 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:58:58.390732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.023873 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:00.023842 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.132312 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:00.132268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:01.719313 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:01.719280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:01.746086 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:01.746059 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 19:59:01.849411 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:01.849375 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.643036 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:02.643001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:03.327016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:03.326979 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.568537 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:04.561603 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.246439 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:06.246406 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.982196 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:07.982161 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.573788 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:09.573760 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.174522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:11.174466 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:12.791919 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:12.791889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.390346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:14.390303 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.035060 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:16.035026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.122399 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:59:16.122359 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:59:17.596061 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:17.595995 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.192180 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:19.192144 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.894524 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:20.894304 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.117016 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:23.115352 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.561618 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:23.561388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.718380 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:23.718345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:24.530522 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:24.527006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:25.150942 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.150901 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.196658 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.196621 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.247692 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.237601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:25.291681 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.291633 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.344634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.344590 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.366628 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.366582 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.389709 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.389669 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.390526 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.390473 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.598577 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.597595 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.868575 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:25.867646 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:26.003076 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:26.003030 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:27.470842 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:27.470805 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.613946 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:27.613827 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.747340 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:27.747257 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.487617 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:29.487583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.934656 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:29.934614 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:30.122634 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:30.122602 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.166716 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:31.166685 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.371520 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:31.371152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.481620 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:31.481547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.206276 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:33.206241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.881700 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:34.881667 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.186771 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.186743 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:59:36.199389 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.199367 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 19:59:36.199799 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.199775 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:59:36.305611 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.305551 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.474734 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.474644 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.597517 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:36.597203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.172162 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:38.172046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:39.128117 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.128086 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7062-6476e4"] Oct 30 19:59:39.134583 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.134561 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:39.155989 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.155937 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7062-6476e4"] Oct 30 19:59:39.158255 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.158228 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\"" Oct 30 19:59:39.192739 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.192697 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5v4\" (UniqueName: \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\") pod \"test-curl-7062-6476e4\" (UID: \"3538e629-e0c8-4a3d-9b11-10de2eb980fd\") " pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:39.294251 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.294213 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5v4\" (UniqueName: \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\") pod \"test-curl-7062-6476e4\" (UID: \"3538e629-e0c8-4a3d-9b11-10de2eb980fd\") " pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:39.315108 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.315075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5v4\" (UniqueName: \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\") pod \"test-curl-7062-6476e4\" (UID: \"3538e629-e0c8-4a3d-9b11-10de2eb980fd\") " pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:39.458131 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.458083 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:39.767178 ip-10-0-132-115 kubenswrapper[4959]: W1030 19:59:39.767147 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3538e629_e0c8_4a3d_9b11_10de2eb980fd.slice/crio-5d5cb622b1fb700323d0c67ddb18a2b2bf6eaf60bf1016241e66336fb29709fe WatchSource:0}: Error finding container 5d5cb622b1fb700323d0c67ddb18a2b2bf6eaf60bf1016241e66336fb29709fe: Status 404 returned error can't find the container with id 5d5cb622b1fb700323d0c67ddb18a2b2bf6eaf60bf1016241e66336fb29709fe Oct 30 19:59:39.769080 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.769034 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7062-6476e4"] Oct 30 19:59:39.816070 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:39.816040 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7062-6476e4" event={"ID":"3538e629-e0c8-4a3d-9b11-10de2eb980fd","Type":"ContainerStarted","Data":"5d5cb622b1fb700323d0c67ddb18a2b2bf6eaf60bf1016241e66336fb29709fe"} Oct 30 19:59:40.221236 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:40.221203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:41.201046 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.201005 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" podUID="38c64900-8ff4-4480-a247-80aeb02e9287" containerName="payload-pre-processing" containerID="cri-o://b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f" gracePeriod=25 Oct 30 19:59:41.594163 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.594126 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:59:41.641390 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.641360 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\") pod \"38c64900-8ff4-4480-a247-80aeb02e9287\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " Oct 30 19:59:41.641904 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.641871 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume" pod "38c64900-8ff4-4480-a247-80aeb02e9287" (UID: "38c64900-8ff4-4480-a247-80aeb02e9287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:41.642052 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.641918 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\" (UniqueName: \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\") pod \"38c64900-8ff4-4480-a247-80aeb02e9287\" (UID: \"38c64900-8ff4-4480-a247-80aeb02e9287\") " Oct 30 19:59:41.643483 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.642848 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c64900-8ff4-4480-a247-80aeb02e9287-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:59:41.647096 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.647055 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb" pod "38c64900-8ff4-4480-a247-80aeb02e9287" (UID: "38c64900-8ff4-4480-a247-80aeb02e9287"). InnerVolumeSpecName "kube-api-access-6dtbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:41.745597 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.745562 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtbb\" (UniqueName: \"kubernetes.io/projected/38c64900-8ff4-4480-a247-80aeb02e9287-kube-api-access-6dtbb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:59:41.848353 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.848273 4959 generic.go:356] "Generic (PLEG): container finished" podID="38c64900-8ff4-4480-a247-80aeb02e9287" containerID="b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f" exitCode=0 Oct 30 19:59:41.848681 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.848332 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" Oct 30 19:59:41.848818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.848356 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" event={"ID":"38c64900-8ff4-4480-a247-80aeb02e9287","Type":"ContainerDied","Data":"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f"} Oct 30 19:59:41.848818 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.848806 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8" event={"ID":"38c64900-8ff4-4480-a247-80aeb02e9287","Type":"ContainerDied","Data":"5aa52330c73613071febd2b29c219f2e88ce519ce9f741db2cf50f8a66dc8242"} Oct 30 19:59:41.848945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.848848 4959 scope.go:122] "RemoveContainer" containerID="b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f" Oct 30 19:59:41.865910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.865846 4959 scope.go:122] "RemoveContainer" containerID="b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f" Oct 30 19:59:41.866768 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:59:41.866740 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f\": container with ID starting with b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f not found: ID does not exist" containerID="b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f" Oct 30 19:59:41.866861 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.866776 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f"} err="failed to get container status \"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f\": rpc error: code = NotFound desc = could not find container \"b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f\": container with ID starting with b8bd85f9bc57661857054b15cfcb79d5dbc9d8891d98b95ece9038795e9d8f3f not found: ID does not exist" Oct 30 19:59:41.870298 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.870276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:41.910410 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.910376 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:59:41.915346 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:41.915317 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d97a5-6fd4bcd45f-rf8l8"] Oct 30 19:59:42.001945 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:42.001912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.005921 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:42.005890 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c64900-8ff4-4480-a247-80aeb02e9287" path="/var/lib/kubelet/pods/38c64900-8ff4-4480-a247-80aeb02e9287/volumes" Oct 30 19:59:42.124579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:42.118335 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.860367 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:42.860280 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7062-6476e4" event={"ID":"3538e629-e0c8-4a3d-9b11-10de2eb980fd","Type":"ContainerStarted","Data":"45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e"} Oct 30 19:59:42.900165 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:42.900111 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7062-6476e4" podStartSLOduration=1.33381359 podStartE2EDuration="3.900095491s" totalImagesPullingTime="2.566281901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:42.8993438 +0000 UTC m=+1507.028577724" watchObservedRunningTime="2026-10-30 19:59:42.900095491 +0000 UTC m=+1507.029329412" Oct 30 19:59:43.624288 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:43.624257 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:43.796443 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:43.796392 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7062-6476e4"] Oct 30 19:59:43.800883 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:43.800856 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7062-6476e4"] Oct 30 19:59:43.878486 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:43.878412 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.875639 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:44.875598 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7062-6476e4" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" containerName="test-curl-7062-6476e4" containerID="cri-o://45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e" gracePeriod=2 Oct 30 19:59:45.178028 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:45.177999 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.724569 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:45.724532 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.066606 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:59:46.066489 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 19:59:47.120150 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.120124 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:47.244579 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.244551 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\" (UniqueName: \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\") pod \"3538e629-e0c8-4a3d-9b11-10de2eb980fd\" (UID: \"3538e629-e0c8-4a3d-9b11-10de2eb980fd\") " Oct 30 19:59:47.246978 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.246947 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4" pod "3538e629-e0c8-4a3d-9b11-10de2eb980fd" (UID: "3538e629-e0c8-4a3d-9b11-10de2eb980fd"). InnerVolumeSpecName "kube-api-access-jb5v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:47.346951 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.346922 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb5v4\" (UniqueName: \"kubernetes.io/projected/3538e629-e0c8-4a3d-9b11-10de2eb980fd-kube-api-access-jb5v4\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 19:59:47.512327 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.512218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:47.904032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.903996 4959 generic.go:356] "Generic (PLEG): container finished" podID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" containerID="45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e" exitCode=137 Oct 30 19:59:47.904032 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.904031 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:47.904280 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.904069 4959 scope.go:122] "RemoveContainer" containerID="45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e" Oct 30 19:59:47.915011 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.914991 4959 scope.go:122] "RemoveContainer" containerID="45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e" Oct 30 19:59:47.915276 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:59:47.915255 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e\": container with ID starting with 45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e not found: ID does not exist" containerID="45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e" Oct 30 19:59:47.915330 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:47.915287 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e"} err="failed to get container status \"45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e\": rpc error: code = NotFound desc = could not find container \"45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e\": container with ID starting with 45981b2704dcd7c64f9cf6babfe6d7b7d93de27a057b103dd69d08e0a462ba6e not found: ID does not exist" Oct 30 19:59:47.927165 ip-10-0-132-115 kubenswrapper[4959]: E1030 19:59:47.927137 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7062-6476e4\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" pod="opendatahub/test-curl-7062-6476e4" Oct 30 19:59:48.002472 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:48.002444 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" path="/var/lib/kubelet/pods/3538e629-e0c8-4a3d-9b11-10de2eb980fd/volumes" Oct 30 19:59:49.273118 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:49.273087 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.282807 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:51.282774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:53.307775 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:53.307741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:54.529413 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:54.529370 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 19:59:54.562720 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:54.562685 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 19:59:54.648017 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:54.647985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:54.775933 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:54.762180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.576466 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:55.576435 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 19:59:55.605440 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:55.605410 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 19:59:55.650643 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:55.650615 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.764152 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:55.764117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:57.771210 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:57.771180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.474844 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:59.474812 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.555910 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:59.555855 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" containerID="cri-o://4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b" gracePeriod=25 Oct 30 19:59:59.580754 ip-10-0-132-115 kubenswrapper[4959]: I1030 19:59:59.580714 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" containerID="cri-o://24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0" gracePeriod=25 Oct 30 20:00:00.245648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.245619 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:00.247012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.246968 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" containerName="test-curl-7062-6476e4" Oct 30 20:00:00.247012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.246995 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" containerName="test-curl-7062-6476e4" Oct 30 20:00:00.247341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.247078 4959 cpu_manager.go:414] "containerMap: removing container" podUID="38c64900-8ff4-4480-a247-80aeb02e9287" containerName="payload-pre-processing" Oct 30 20:00:00.247341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.247087 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c64900-8ff4-4480-a247-80aeb02e9287" containerName="payload-pre-processing" Oct 30 20:00:00.247341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.247302 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c64900-8ff4-4480-a247-80aeb02e9287" containerName="payload-pre-processing" Oct 30 20:00:00.247341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.247327 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3538e629-e0c8-4a3d-9b11-10de2eb980fd" containerName="test-curl-7062-6476e4" Oct 30 20:00:00.251549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.251531 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.267877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.267850 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:00.274660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.274290 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.276593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.276569 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:00.281408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.281385 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:00.285929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.285799 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:00:00.287634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.287615 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.301635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.290081 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:00.311898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.311865 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvwx6\" (UniqueName: \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.312360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.312334 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.322789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.321809 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:00.327635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.327616 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.328382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.328363 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:00.335601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.334432 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:00.342127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.342089 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.349214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.349193 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:00.384544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.384471 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:00.414085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414053 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.414295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414111 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchgb\" (UniqueName: \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.414295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414180 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.414295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414259 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414390 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbsz\" (UniqueName: \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414437 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414551 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c72q\" (UniqueName: \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414611 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkz9\" (UniqueName: \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414651 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvwx6\" (UniqueName: \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.414881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.414838 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.415379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.415359 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.439442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.439414 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvwx6\" (UniqueName: \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.516490 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516419 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c72q\" (UniqueName: \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.516490 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516470 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkz9\" (UniqueName: \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516637 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516671 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516700 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchgb\" (UniqueName: \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516775 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516899 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbsz\" (UniqueName: \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.517170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.516946 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.517420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.517399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.517561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.517441 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.517561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.517441 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.517717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.517695 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.543703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.543677 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchgb\" (UniqueName: \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\") pod \"maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.543887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.543714 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbsz\" (UniqueName: \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.543963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.543938 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c72q\" (UniqueName: \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.549864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.549837 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkz9\" (UniqueName: \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.577765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.577734 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" podUID="9fb02c7b-6156-4254-9d1c-778d372ca467" containerName="payload-processing" containerID="cri-o://7b737274848ccac997cc298e64884294607e091fd20864163120c4b2db5e75d0" gracePeriod=25 Oct 30 20:00:00.583045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.583025 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:00.606205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.606165 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" podUID="353e7a62-4777-4375-b4d7-a06caf2940e1" containerName="payload-pre-processing" containerID="cri-o://423e989ded65f27e193c1acfa1fb5a43872efb98b4bdbe7d1ff31c0c9f29585f" gracePeriod=25 Oct 30 20:00:00.626910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.626884 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:00.629717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.629696 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:00.654304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.654277 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:00.672272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.670847 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:00.907959 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:00.907929 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9701ae_3658_4c5e_bb57_9d604643d65f.slice/crio-8d5a81edd2292431e3f11c663919a22a50a8568c599def2fd8a3567ab5e911d6 WatchSource:0}: Error finding container 8d5a81edd2292431e3f11c663919a22a50a8568c599def2fd8a3567ab5e911d6: Status 404 returned error can't find the container with id 8d5a81edd2292431e3f11c663919a22a50a8568c599def2fd8a3567ab5e911d6 Oct 30 20:00:00.909064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:00.909041 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:01.061123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.060971 4959 generic.go:356] "Generic (PLEG): container finished" podID="353e7a62-4777-4375-b4d7-a06caf2940e1" containerID="423e989ded65f27e193c1acfa1fb5a43872efb98b4bdbe7d1ff31c0c9f29585f" exitCode=0 Oct 30 20:00:01.061320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.061140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" event={"ID":"353e7a62-4777-4375-b4d7-a06caf2940e1","Type":"ContainerDied","Data":"423e989ded65f27e193c1acfa1fb5a43872efb98b4bdbe7d1ff31c0c9f29585f"} Oct 30 20:00:01.063396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.063373 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerStarted","Data":"8d5a81edd2292431e3f11c663919a22a50a8568c599def2fd8a3567ab5e911d6"} Oct 30 20:00:01.065304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.065272 4959 generic.go:356] "Generic (PLEG): container finished" podID="9fb02c7b-6156-4254-9d1c-778d372ca467" containerID="7b737274848ccac997cc298e64884294607e091fd20864163120c4b2db5e75d0" exitCode=0 Oct 30 20:00:01.065421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.065343 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" event={"ID":"9fb02c7b-6156-4254-9d1c-778d372ca467","Type":"ContainerDied","Data":"7b737274848ccac997cc298e64884294607e091fd20864163120c4b2db5e75d0"} Oct 30 20:00:01.162593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.162566 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 20:00:01.166034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.166013 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 20:00:01.189246 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:01.189218 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638a7e34_94d9_4209_8562_2fb90d848609.slice/crio-9bf2f85ed49b6e4cd3c66d9e2817b65544cba9e3dc79cf96da21b89cfd46a00c WatchSource:0}: Error finding container 9bf2f85ed49b6e4cd3c66d9e2817b65544cba9e3dc79cf96da21b89cfd46a00c: Status 404 returned error can't find the container with id 9bf2f85ed49b6e4cd3c66d9e2817b65544cba9e3dc79cf96da21b89cfd46a00c Oct 30 20:00:01.189462 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.189442 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:01.205805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.205743 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:01.208677 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:01.208648 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc23b796_bbd4_4b71_bee4_66e7cea06c97.slice/crio-dfe1ea81b41340624881d8fe3c6b36e3ca8f714b83b03d9364de84b9fd168ab9 WatchSource:0}: Error finding container dfe1ea81b41340624881d8fe3c6b36e3ca8f714b83b03d9364de84b9fd168ab9: Status 404 returned error can't find the container with id dfe1ea81b41340624881d8fe3c6b36e3ca8f714b83b03d9364de84b9fd168ab9 Oct 30 20:00:01.234830 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.234795 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:01.239900 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:01.239876 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ad6203_4582_432c_9ef0_b8efab87118e.slice/crio-76a53b19558132cccb7177c77b6dc1080b24110cde60e5c1cefc7b6a1b6ac5c6 WatchSource:0}: Error finding container 76a53b19558132cccb7177c77b6dc1080b24110cde60e5c1cefc7b6a1b6ac5c6: Status 404 returned error can't find the container with id 76a53b19558132cccb7177c77b6dc1080b24110cde60e5c1cefc7b6a1b6ac5c6 Oct 30 20:00:01.243945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.243922 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\") pod \"353e7a62-4777-4375-b4d7-a06caf2940e1\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " Oct 30 20:00:01.246126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.244034 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\" (UniqueName: \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\") pod \"353e7a62-4777-4375-b4d7-a06caf2940e1\" (UID: \"353e7a62-4777-4375-b4d7-a06caf2940e1\") " Oct 30 20:00:01.246126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.244145 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\") pod \"9fb02c7b-6156-4254-9d1c-778d372ca467\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " Oct 30 20:00:01.246126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.244198 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\" (UniqueName: \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\") pod \"9fb02c7b-6156-4254-9d1c-778d372ca467\" (UID: \"9fb02c7b-6156-4254-9d1c-778d372ca467\") " Oct 30 20:00:01.246126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.244476 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume" pod "353e7a62-4777-4375-b4d7-a06caf2940e1" (UID: "353e7a62-4777-4375-b4d7-a06caf2940e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:01.246126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.245174 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353e7a62-4777-4375-b4d7-a06caf2940e1-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:01.247415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.247392 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg" pod "9fb02c7b-6156-4254-9d1c-778d372ca467" (UID: "9fb02c7b-6156-4254-9d1c-778d372ca467"). InnerVolumeSpecName "kube-api-access-brzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:01.247776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.247756 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume" pod "9fb02c7b-6156-4254-9d1c-778d372ca467" (UID: "9fb02c7b-6156-4254-9d1c-778d372ca467"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:01.249162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.249125 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl" pod "353e7a62-4777-4375-b4d7-a06caf2940e1" (UID: "353e7a62-4777-4375-b4d7-a06caf2940e1"). InnerVolumeSpecName "kube-api-access-l8lbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:01.295785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.295752 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:01.347725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.347649 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lbl\" (UniqueName: \"kubernetes.io/projected/353e7a62-4777-4375-b4d7-a06caf2940e1-kube-api-access-l8lbl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:01.347725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.347688 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fb02c7b-6156-4254-9d1c-778d372ca467-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:01.347725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.347704 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzqg\" (UniqueName: \"kubernetes.io/projected/9fb02c7b-6156-4254-9d1c-778d372ca467-kube-api-access-brzqg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:01.353959 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:01.353918 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode167265d_4b6a_436a_8514_ab795d341fde.slice/crio-727b418be8c0c5645db40cf3e98155344867aecbe374a5b3c30cd16068857597 WatchSource:0}: Error finding container 727b418be8c0c5645db40cf3e98155344867aecbe374a5b3c30cd16068857597: Status 404 returned error can't find the container with id 727b418be8c0c5645db40cf3e98155344867aecbe374a5b3c30cd16068857597 Oct 30 20:00:01.533276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:01.533173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:02.103131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.103083 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerStarted","Data":"792d1210d98912aa0825ab13325afdf3e8e819d889973b94db3349c025906dcc"} Oct 30 20:00:02.103298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.103149 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerStarted","Data":"dfe1ea81b41340624881d8fe3c6b36e3ca8f714b83b03d9364de84b9fd168ab9"} Oct 30 20:00:02.105200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.105170 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerStarted","Data":"eff45d99b056e05d0731cdb451110b7bbbf7b5cd494cb001021ea93c44a1f40b"} Oct 30 20:00:02.105326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.105223 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerStarted","Data":"9bf2f85ed49b6e4cd3c66d9e2817b65544cba9e3dc79cf96da21b89cfd46a00c"} Oct 30 20:00:02.107117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.107095 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerStarted","Data":"c5c8d239cdbeadaa5402b00e9840dfaa3043d002cd22f4d20dbf952326968e46"} Oct 30 20:00:02.107219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.107136 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerStarted","Data":"76a53b19558132cccb7177c77b6dc1080b24110cde60e5c1cefc7b6a1b6ac5c6"} Oct 30 20:00:02.108933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.108909 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerStarted","Data":"727b418be8c0c5645db40cf3e98155344867aecbe374a5b3c30cd16068857597"} Oct 30 20:00:02.110455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.110434 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" event={"ID":"353e7a62-4777-4375-b4d7-a06caf2940e1","Type":"ContainerDied","Data":"a7885afa77863ce118e55912fddbe818f7ecc969f5c37b28336db5bec4721642"} Oct 30 20:00:02.110618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.110471 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx" Oct 30 20:00:02.110618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.110486 4959 scope.go:122] "RemoveContainer" containerID="423e989ded65f27e193c1acfa1fb5a43872efb98b4bdbe7d1ff31c0c9f29585f" Oct 30 20:00:02.112309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.112284 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerStarted","Data":"f9b94cc5b8ef77e1a0258244b794cf5db8579351205c979775d90f2653706d17"} Oct 30 20:00:02.113956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.113932 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" event={"ID":"9fb02c7b-6156-4254-9d1c-778d372ca467","Type":"ContainerDied","Data":"2d2ac8dca3c73852545a95771394b16de997d79820db31aa55a07eaa4fdf6580"} Oct 30 20:00:02.114056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.114037 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5" Oct 30 20:00:02.126018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.125993 4959 scope.go:122] "RemoveContainer" containerID="7b737274848ccac997cc298e64884294607e091fd20864163120c4b2db5e75d0" Oct 30 20:00:02.149114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.149065 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" podStartSLOduration=2.149050617 podStartE2EDuration="2.149050617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.144155916 +0000 UTC m=+1526.273389837" watchObservedRunningTime="2026-10-30 20:00:02.149050617 +0000 UTC m=+1526.278284543" Oct 30 20:00:02.188828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.188772 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" podStartSLOduration=2.188757394 podStartE2EDuration="2.188757394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.187126059 +0000 UTC m=+1526.316359993" watchObservedRunningTime="2026-10-30 20:00:02.188757394 +0000 UTC m=+1526.317991314" Oct 30 20:00:02.231347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.231289 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" podStartSLOduration=2.231273048 podStartE2EDuration="2.231273048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.229016833 +0000 UTC m=+1526.358250751" watchObservedRunningTime="2026-10-30 20:00:02.231273048 +0000 UTC m=+1526.360507165" Oct 30 20:00:02.272754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.272723 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 20:00:02.277631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.277602 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-93263c-78d9f79dbc-dgwqx"] Oct 30 20:00:02.318151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.318054 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" podStartSLOduration=2.318040838 podStartE2EDuration="2.318040838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.313194726 +0000 UTC m=+1526.442428650" watchObservedRunningTime="2026-10-30 20:00:02.318040838 +0000 UTC m=+1526.447274796" Oct 30 20:00:02.357271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.357221 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" podStartSLOduration=2.357206046 podStartE2EDuration="2.357206046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.355248363 +0000 UTC m=+1526.484482284" watchObservedRunningTime="2026-10-30 20:00:02.357206046 +0000 UTC m=+1526.486439966" Oct 30 20:00:02.399755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.399713 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 20:00:02.405168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:02.405142 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-93263c-7fb7b9d6c8-2czs5"] Oct 30 20:00:03.125758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.125669 4959 generic.go:356] "Generic (PLEG): container finished" podID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerID="f9b94cc5b8ef77e1a0258244b794cf5db8579351205c979775d90f2653706d17" exitCode=60 Oct 30 20:00:03.126148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.125792 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerDied","Data":"f9b94cc5b8ef77e1a0258244b794cf5db8579351205c979775d90f2653706d17"} Oct 30 20:00:03.126223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.126207 4959 scope.go:122] "RemoveContainer" containerID="f9b94cc5b8ef77e1a0258244b794cf5db8579351205c979775d90f2653706d17" Oct 30 20:00:03.130048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.130016 4959 generic.go:356] "Generic (PLEG): container finished" podID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerID="792d1210d98912aa0825ab13325afdf3e8e819d889973b94db3349c025906dcc" exitCode=60 Oct 30 20:00:03.130187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.130123 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerDied","Data":"792d1210d98912aa0825ab13325afdf3e8e819d889973b94db3349c025906dcc"} Oct 30 20:00:03.130377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.130367 4959 scope.go:122] "RemoveContainer" containerID="792d1210d98912aa0825ab13325afdf3e8e819d889973b94db3349c025906dcc" Oct 30 20:00:03.131961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.131935 4959 generic.go:356] "Generic (PLEG): container finished" podID="638a7e34-94d9-4209-8562-2fb90d848609" containerID="eff45d99b056e05d0731cdb451110b7bbbf7b5cd494cb001021ea93c44a1f40b" exitCode=60 Oct 30 20:00:03.132098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.132022 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerDied","Data":"eff45d99b056e05d0731cdb451110b7bbbf7b5cd494cb001021ea93c44a1f40b"} Oct 30 20:00:03.132313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.132296 4959 scope.go:122] "RemoveContainer" containerID="eff45d99b056e05d0731cdb451110b7bbbf7b5cd494cb001021ea93c44a1f40b" Oct 30 20:00:03.133750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.133730 4959 generic.go:356] "Generic (PLEG): container finished" podID="13ad6203-4582-432c-9ef0-b8efab87118e" containerID="c5c8d239cdbeadaa5402b00e9840dfaa3043d002cd22f4d20dbf952326968e46" exitCode=60 Oct 30 20:00:03.133851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.133798 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerDied","Data":"c5c8d239cdbeadaa5402b00e9840dfaa3043d002cd22f4d20dbf952326968e46"} Oct 30 20:00:03.134008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.133993 4959 scope.go:122] "RemoveContainer" containerID="c5c8d239cdbeadaa5402b00e9840dfaa3043d002cd22f4d20dbf952326968e46" Oct 30 20:00:03.135529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.135506 4959 generic.go:356] "Generic (PLEG): container finished" podID="e167265d-4b6a-436a-8514-ab795d341fde" containerID="3275bee161874312e3e9a3a7c7f66e97ba9373624293cd347669ba02c40c1aa0" exitCode=60 Oct 30 20:00:03.135625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.135561 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerDied","Data":"3275bee161874312e3e9a3a7c7f66e97ba9373624293cd347669ba02c40c1aa0"} Oct 30 20:00:03.135854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.135840 4959 scope.go:122] "RemoveContainer" containerID="3275bee161874312e3e9a3a7c7f66e97ba9373624293cd347669ba02c40c1aa0" Oct 30 20:00:03.180104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:03.179992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.007949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.007876 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353e7a62-4777-4375-b4d7-a06caf2940e1" path="/var/lib/kubelet/pods/353e7a62-4777-4375-b4d7-a06caf2940e1/volumes" Oct 30 20:00:04.008337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.008322 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb02c7b-6156-4254-9d1c-778d372ca467" path="/var/lib/kubelet/pods/9fb02c7b-6156-4254-9d1c-778d372ca467/volumes" Oct 30 20:00:04.146271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.146192 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerStarted","Data":"e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd"} Oct 30 20:00:04.155514 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.155411 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerStarted","Data":"6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945"} Oct 30 20:00:04.172650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.172612 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerStarted","Data":"52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104"} Oct 30 20:00:04.180102 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.180067 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerStarted","Data":"b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8"} Oct 30 20:00:04.182747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.182719 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerStarted","Data":"501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267"} Oct 30 20:00:04.966578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:04.949699 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.193747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.193715 4959 generic.go:356] "Generic (PLEG): container finished" podID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerID="e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd" exitCode=60 Oct 30 20:00:05.194208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.193805 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerDied","Data":"e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd"} Oct 30 20:00:05.194208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.193866 4959 scope.go:122] "RemoveContainer" containerID="792d1210d98912aa0825ab13325afdf3e8e819d889973b94db3349c025906dcc" Oct 30 20:00:05.194208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.194151 4959 scope.go:122] "RemoveContainer" containerID="e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd" Oct 30 20:00:05.194636 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:05.194612 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq_odh-ai-gateway-infra(fc23b796-bbd4-4b71-bee4-66e7cea06c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" Oct 30 20:00:05.196095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.196075 4959 generic.go:356] "Generic (PLEG): container finished" podID="638a7e34-94d9-4209-8562-2fb90d848609" containerID="6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945" exitCode=60 Oct 30 20:00:05.196199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.196155 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerDied","Data":"6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945"} Oct 30 20:00:05.196504 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.196474 4959 scope.go:122] "RemoveContainer" containerID="6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945" Oct 30 20:00:05.196911 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:05.196889 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm_odh-ai-gateway-infra(638a7e34-94d9-4209-8562-2fb90d848609)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" podUID="638a7e34-94d9-4209-8562-2fb90d848609" Oct 30 20:00:05.198239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.198217 4959 generic.go:356] "Generic (PLEG): container finished" podID="13ad6203-4582-432c-9ef0-b8efab87118e" containerID="52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104" exitCode=60 Oct 30 20:00:05.198328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.198267 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerDied","Data":"52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104"} Oct 30 20:00:05.198625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.198605 4959 scope.go:122] "RemoveContainer" containerID="52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104" Oct 30 20:00:05.199003 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:05.198983 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj_odh-ai-gateway-infra(13ad6203-4582-432c-9ef0-b8efab87118e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" Oct 30 20:00:05.200770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.200747 4959 generic.go:356] "Generic (PLEG): container finished" podID="e167265d-4b6a-436a-8514-ab795d341fde" containerID="b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8" exitCode=60 Oct 30 20:00:05.201168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.200880 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerDied","Data":"b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8"} Oct 30 20:00:05.201168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.201122 4959 scope.go:122] "RemoveContainer" containerID="b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8" Oct 30 20:00:05.201533 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:05.201469 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2_odh-ai-gateway-infra(e167265d-4b6a-436a-8514-ab795d341fde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" podUID="e167265d-4b6a-436a-8514-ab795d341fde" Oct 30 20:00:05.203395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.203361 4959 generic.go:356] "Generic (PLEG): container finished" podID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerID="501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267" exitCode=60 Oct 30 20:00:05.203549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.203413 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerDied","Data":"501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267"} Oct 30 20:00:05.203807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.203789 4959 scope.go:122] "RemoveContainer" containerID="501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267" Oct 30 20:00:05.206107 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:05.204207 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj_odh-ai-gateway-infra(4b9701ae-3658-4c5e-bb57-9d604643d65f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" Oct 30 20:00:05.211782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.211761 4959 scope.go:122] "RemoveContainer" containerID="eff45d99b056e05d0731cdb451110b7bbbf7b5cd494cb001021ea93c44a1f40b" Oct 30 20:00:05.227218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.227196 4959 scope.go:122] "RemoveContainer" containerID="c5c8d239cdbeadaa5402b00e9840dfaa3043d002cd22f4d20dbf952326968e46" Oct 30 20:00:05.241658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.241638 4959 scope.go:122] "RemoveContainer" containerID="3275bee161874312e3e9a3a7c7f66e97ba9373624293cd347669ba02c40c1aa0" Oct 30 20:00:05.256489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:05.256467 4959 scope.go:122] "RemoveContainer" containerID="f9b94cc5b8ef77e1a0258244b794cf5db8579351205c979775d90f2653706d17" Oct 30 20:00:06.213226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.213193 4959 scope.go:122] "RemoveContainer" containerID="52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104" Oct 30 20:00:06.213724 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:06.213695 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj_odh-ai-gateway-infra(13ad6203-4582-432c-9ef0-b8efab87118e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" Oct 30 20:00:06.214941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.214913 4959 scope.go:122] "RemoveContainer" containerID="b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8" Oct 30 20:00:06.215287 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:06.215263 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2_odh-ai-gateway-infra(e167265d-4b6a-436a-8514-ab795d341fde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" podUID="e167265d-4b6a-436a-8514-ab795d341fde" Oct 30 20:00:06.216643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.216624 4959 scope.go:122] "RemoveContainer" containerID="501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267" Oct 30 20:00:06.217004 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:06.216980 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj_odh-ai-gateway-infra(4b9701ae-3658-4c5e-bb57-9d604643d65f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" Oct 30 20:00:06.218255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.218236 4959 scope.go:122] "RemoveContainer" containerID="e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd" Oct 30 20:00:06.218558 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:06.218538 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq_odh-ai-gateway-infra(fc23b796-bbd4-4b71-bee4-66e7cea06c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" Oct 30 20:00:06.219862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.219846 4959 scope.go:122] "RemoveContainer" containerID="6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945" Oct 30 20:00:06.220092 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:06.220075 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm_odh-ai-gateway-infra(638a7e34-94d9-4209-8562-2fb90d848609)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" podUID="638a7e34-94d9-4209-8562-2fb90d848609" Oct 30 20:00:06.706005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:06.705970 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:08.223229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:08.223191 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:08.223711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:08.223462 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:08.706083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:08.706045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.501516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:10.495152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.200915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:12.200884 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.326586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:12.326558 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.901801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:12.901770 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:13.378362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:13.378273 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.246742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:14.246683 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.714970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:14.714940 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.994412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:15.994383 4959 scope.go:122] "RemoveContainer" containerID="e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd" Oct 30 20:00:16.074784 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:16.074750 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:00:16.569436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.569411 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.600945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.600921 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerStarted","Data":"f22b8f261690c48302aa6ccd0b8b1760fd737b95caedc73a8b3fbb841a15bccf"} Oct 30 20:00:16.989845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.989410 4959 scope.go:122] "RemoveContainer" containerID="b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8" Oct 30 20:00:16.990074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.989933 4959 scope.go:122] "RemoveContainer" containerID="52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104" Oct 30 20:00:16.990146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.990107 4959 scope.go:122] "RemoveContainer" containerID="6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945" Oct 30 20:00:16.990436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:16.990413 4959 scope.go:122] "RemoveContainer" containerID="501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267" Oct 30 20:00:17.329364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.329327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.504961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.504932 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.610681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.610651 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerStarted","Data":"cdaba02e7b5bac5cf742ef4f424f2396cbc6e882be27f43af801174472b9d890"} Oct 30 20:00:17.612543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.612517 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerStarted","Data":"5544dedcae4d28a539ce1c24612ee1e9143b219809e496302857beb3f42e11c7"} Oct 30 20:00:17.614410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.614389 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerStarted","Data":"955bded5c3a5a62876ecf41dae144d784532828efb818e7fd6edcaee828f52db"} Oct 30 20:00:17.616348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.616329 4959 generic.go:356] "Generic (PLEG): container finished" podID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerID="f22b8f261690c48302aa6ccd0b8b1760fd737b95caedc73a8b3fbb841a15bccf" exitCode=60 Oct 30 20:00:17.616446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.616402 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerDied","Data":"f22b8f261690c48302aa6ccd0b8b1760fd737b95caedc73a8b3fbb841a15bccf"} Oct 30 20:00:17.616446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.616443 4959 scope.go:122] "RemoveContainer" containerID="e6a327deee6c963e3f41eefd89d890b92b2137645848e85b31848f62065015cd" Oct 30 20:00:17.616769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.616753 4959 scope.go:122] "RemoveContainer" containerID="f22b8f261690c48302aa6ccd0b8b1760fd737b95caedc73a8b3fbb841a15bccf" Oct 30 20:00:17.617078 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:17.617053 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq_odh-ai-gateway-infra(fc23b796-bbd4-4b71-bee4-66e7cea06c97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" Oct 30 20:00:17.618282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.618265 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerStarted","Data":"369a6d017ef2ae7fbd5c204e044acf2116209c6fc3f7869b5fea709c832b2557"} Oct 30 20:00:17.678079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:17.678051 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:18.223169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.223094 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:18.223169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.223139 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:00:18.498398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.498368 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.668529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.664082 4959 generic.go:356] "Generic (PLEG): container finished" podID="13ad6203-4582-432c-9ef0-b8efab87118e" containerID="cdaba02e7b5bac5cf742ef4f424f2396cbc6e882be27f43af801174472b9d890" exitCode=60 Oct 30 20:00:18.668529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.664200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerDied","Data":"cdaba02e7b5bac5cf742ef4f424f2396cbc6e882be27f43af801174472b9d890"} Oct 30 20:00:18.668529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.664276 4959 scope.go:122] "RemoveContainer" containerID="52258de681b3e9993ca4bddbd314f9d2cff82a195c1f26c7dd2221a576bfd104" Oct 30 20:00:18.668529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.664887 4959 scope.go:122] "RemoveContainer" containerID="cdaba02e7b5bac5cf742ef4f424f2396cbc6e882be27f43af801174472b9d890" Oct 30 20:00:18.668529 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:18.665373 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj_odh-ai-gateway-infra(13ad6203-4582-432c-9ef0-b8efab87118e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" Oct 30 20:00:18.680386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.679795 4959 generic.go:356] "Generic (PLEG): container finished" podID="e167265d-4b6a-436a-8514-ab795d341fde" containerID="5544dedcae4d28a539ce1c24612ee1e9143b219809e496302857beb3f42e11c7" exitCode=60 Oct 30 20:00:18.680386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.680008 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerDied","Data":"5544dedcae4d28a539ce1c24612ee1e9143b219809e496302857beb3f42e11c7"} Oct 30 20:00:18.680652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.680411 4959 scope.go:122] "RemoveContainer" containerID="5544dedcae4d28a539ce1c24612ee1e9143b219809e496302857beb3f42e11c7" Oct 30 20:00:18.681221 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:18.680848 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2_odh-ai-gateway-infra(e167265d-4b6a-436a-8514-ab795d341fde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" podUID="e167265d-4b6a-436a-8514-ab795d341fde" Oct 30 20:00:18.683152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.683061 4959 generic.go:356] "Generic (PLEG): container finished" podID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerID="955bded5c3a5a62876ecf41dae144d784532828efb818e7fd6edcaee828f52db" exitCode=60 Oct 30 20:00:18.683427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.683406 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerDied","Data":"955bded5c3a5a62876ecf41dae144d784532828efb818e7fd6edcaee828f52db"} Oct 30 20:00:18.683587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.683573 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:18.684152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.684135 4959 scope.go:122] "RemoveContainer" containerID="955bded5c3a5a62876ecf41dae144d784532828efb818e7fd6edcaee828f52db" Oct 30 20:00:18.684733 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:18.684708 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj_odh-ai-gateway-infra(4b9701ae-3658-4c5e-bb57-9d604643d65f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" Oct 30 20:00:18.689987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.689955 4959 generic.go:356] "Generic (PLEG): container finished" podID="638a7e34-94d9-4209-8562-2fb90d848609" containerID="369a6d017ef2ae7fbd5c204e044acf2116209c6fc3f7869b5fea709c832b2557" exitCode=60 Oct 30 20:00:18.690163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.690016 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerDied","Data":"369a6d017ef2ae7fbd5c204e044acf2116209c6fc3f7869b5fea709c832b2557"} Oct 30 20:00:18.690424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.690409 4959 scope.go:122] "RemoveContainer" containerID="369a6d017ef2ae7fbd5c204e044acf2116209c6fc3f7869b5fea709c832b2557" Oct 30 20:00:18.690939 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:18.690916 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm_odh-ai-gateway-infra(638a7e34-94d9-4209-8562-2fb90d848609)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" podUID="638a7e34-94d9-4209-8562-2fb90d848609" Oct 30 20:00:18.697029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.696986 4959 scope.go:122] "RemoveContainer" containerID="b68ef514657098371b4af681426701f8c3314cce3c1ad7f32f9417779ab80db8" Oct 30 20:00:18.716126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.716105 4959 scope.go:122] "RemoveContainer" containerID="501965e4a5a7d7c4588b30e424db00e43f0f6873bc258267acd9d3da0319e267" Oct 30 20:00:18.728761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.728735 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:18.735787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.735767 4959 scope.go:122] "RemoveContainer" containerID="6630d2fce63ed0df243f94a2d6b0afe96457210dbceaad9732d2b9b94df13945" Oct 30 20:00:18.803780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.802428 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:18.891008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.890973 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:18.902742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.902716 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:18.998224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.998134 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\" (UniqueName: \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\") pod \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " Oct 30 20:00:18.998434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.998416 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\") pod \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\" (UID: \"fc23b796-bbd4-4b71-bee4-66e7cea06c97\") " Oct 30 20:00:18.998741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.998720 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca" pod "fc23b796-bbd4-4b71-bee4-66e7cea06c97" (UID: "fc23b796-bbd4-4b71-bee4-66e7cea06c97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:18.999138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:18.999119 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc23b796-bbd4-4b71-bee4-66e7cea06c97-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.000433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.000410 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb" pod "fc23b796-bbd4-4b71-bee4-66e7cea06c97" (UID: "fc23b796-bbd4-4b71-bee4-66e7cea06c97"). InnerVolumeSpecName "kube-api-access-lchgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.039586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.039522 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.101345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.101316 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchgb\" (UniqueName: \"kubernetes.io/projected/fc23b796-bbd4-4b71-bee4-66e7cea06c97-kube-api-access-lchgb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:19.197068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.197034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.424522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.420621 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.744657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.743787 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" event={"ID":"fc23b796-bbd4-4b71-bee4-66e7cea06c97","Type":"ContainerDied","Data":"dfe1ea81b41340624881d8fe3c6b36e3ca8f714b83b03d9364de84b9fd168ab9"} Oct 30 20:00:19.744657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.743853 4959 scope.go:122] "RemoveContainer" containerID="f22b8f261690c48302aa6ccd0b8b1760fd737b95caedc73a8b3fbb841a15bccf" Oct 30 20:00:19.744657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.743942 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq" Oct 30 20:00:19.990974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:19.990875 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:20.012573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.012535 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d96913-29889840-hfbbq"] Oct 30 20:00:20.096446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.096419 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:20.178038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.178008 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\" (UniqueName: \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\") pod \"13ad6203-4582-432c-9ef0-b8efab87118e\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " Oct 30 20:00:20.178227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.178208 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\") pod \"13ad6203-4582-432c-9ef0-b8efab87118e\" (UID: \"13ad6203-4582-432c-9ef0-b8efab87118e\") " Oct 30 20:00:20.179451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.179422 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca" pod "13ad6203-4582-432c-9ef0-b8efab87118e" (UID: "13ad6203-4582-432c-9ef0-b8efab87118e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.182448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.182417 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9" pod "13ad6203-4582-432c-9ef0-b8efab87118e" (UID: "13ad6203-4582-432c-9ef0-b8efab87118e"). InnerVolumeSpecName "kube-api-access-htkz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.227130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.227102 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 20:00:20.234115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.234090 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:20.283433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.283404 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ad6203-4582-432c-9ef0-b8efab87118e-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.283551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.283440 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkz9\" (UniqueName: \"kubernetes.io/projected/13ad6203-4582-432c-9ef0-b8efab87118e-kube-api-access-htkz9\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.299956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.299931 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:20.309768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.309740 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:20.323239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.322862 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 20:00:20.392437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.392346 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\" (UniqueName: \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\") pod \"4b9701ae-3658-4c5e-bb57-9d604643d65f\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " Oct 30 20:00:20.397337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.392845 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\" (UniqueName: \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\") pod \"638a7e34-94d9-4209-8562-2fb90d848609\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " Oct 30 20:00:20.397337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.392956 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\") pod \"96832826-53f3-4b64-b4aa-5eb28e9dc279\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " Oct 30 20:00:20.397337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.393001 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\") pod \"638a7e34-94d9-4209-8562-2fb90d848609\" (UID: \"638a7e34-94d9-4209-8562-2fb90d848609\") " Oct 30 20:00:20.397337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.393182 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\") pod \"4b9701ae-3658-4c5e-bb57-9d604643d65f\" (UID: \"4b9701ae-3658-4c5e-bb57-9d604643d65f\") " Oct 30 20:00:20.397337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.393232 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\" (UniqueName: \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\") pod \"96832826-53f3-4b64-b4aa-5eb28e9dc279\" (UID: \"96832826-53f3-4b64-b4aa-5eb28e9dc279\") " Oct 30 20:00:20.420569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.418715 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca" pod "638a7e34-94d9-4209-8562-2fb90d848609" (UID: "638a7e34-94d9-4209-8562-2fb90d848609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.436466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.429709 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume" pod "96832826-53f3-4b64-b4aa-5eb28e9dc279" (UID: "96832826-53f3-4b64-b4aa-5eb28e9dc279"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.436466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.430542 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca" pod "4b9701ae-3658-4c5e-bb57-9d604643d65f" (UID: "4b9701ae-3658-4c5e-bb57-9d604643d65f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.439300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.437238 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96832826-53f3-4b64-b4aa-5eb28e9dc279-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.439300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.437280 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/638a7e34-94d9-4209-8562-2fb90d848609-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.439300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.437296 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b9701ae-3658-4c5e-bb57-9d604643d65f-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.460487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.460386 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6" pod "96832826-53f3-4b64-b4aa-5eb28e9dc279" (UID: "96832826-53f3-4b64-b4aa-5eb28e9dc279"). InnerVolumeSpecName "kube-api-access-x4kc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.484155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.484101 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q" pod "638a7e34-94d9-4209-8562-2fb90d848609" (UID: "638a7e34-94d9-4209-8562-2fb90d848609"). InnerVolumeSpecName "kube-api-access-9c72q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.485218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.485175 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6" pod "4b9701ae-3658-4c5e-bb57-9d604643d65f" (UID: "4b9701ae-3658-4c5e-bb57-9d604643d65f"). InnerVolumeSpecName "kube-api-access-tvwx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.488864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.488834 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-4925e2"] Oct 30 20:00:20.490195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490170 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.490195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490196 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490225 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490232 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490254 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490262 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490276 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9fb02c7b-6156-4254-9d1c-778d372ca467" containerName="payload-processing" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490282 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb02c7b-6156-4254-9d1c-778d372ca467" containerName="payload-processing" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490302 4959 cpu_manager.go:414] "containerMap: removing container" podUID="353e7a62-4777-4375-b4d7-a06caf2940e1" containerName="payload-pre-processing" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490309 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353e7a62-4777-4375-b4d7-a06caf2940e1" containerName="payload-pre-processing" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490322 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490328 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490341 4959 cpu_manager.go:414] "containerMap: removing container" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490348 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.490353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490358 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490364 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490372 4959 cpu_manager.go:414] "containerMap: removing container" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490380 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490392 4959 cpu_manager.go:414] "containerMap: removing container" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490398 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490411 4959 cpu_manager.go:414] "containerMap: removing container" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490417 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490428 4959 cpu_manager.go:414] "containerMap: removing container" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490434 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490443 4959 cpu_manager.go:414] "containerMap: removing container" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490454 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490465 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490471 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490484 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490632 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490652 4959 cpu_manager.go:414] "containerMap: removing container" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490659 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490668 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490674 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490692 4959 cpu_manager.go:414] "containerMap: removing container" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490699 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490936 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490952 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490962 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb02c7b-6156-4254-9d1c-778d372ca467" containerName="payload-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490975 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="353e7a62-4777-4375-b4d7-a06caf2940e1" containerName="payload-pre-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490983 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e167265d-4b6a-436a-8514-ab795d341fde" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490992 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.490999 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491006 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491015 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491024 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491031 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491044 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491052 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491059 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491068 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerName="payload-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491077 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" containerName="cleanup" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491086 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerName="payload-pre-processing" Oct 30 20:00:20.491114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.491098 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:20.496115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.496095 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:20.501289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.500612 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-4925e2"] Oct 30 20:00:20.514890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.514850 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\"" Oct 30 20:00:20.538427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.538383 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\") pod \"e167265d-4b6a-436a-8514-ab795d341fde\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " Oct 30 20:00:20.538602 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.538513 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\" (UniqueName: \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\") pod \"07f74d5e-eeab-410d-8a8e-0dadf151c738\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " Oct 30 20:00:20.538660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.538642 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\") pod \"07f74d5e-eeab-410d-8a8e-0dadf151c738\" (UID: \"07f74d5e-eeab-410d-8a8e-0dadf151c738\") " Oct 30 20:00:20.538920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.538897 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca" pod "e167265d-4b6a-436a-8514-ab795d341fde" (UID: "e167265d-4b6a-436a-8514-ab795d341fde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.539018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.538999 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\" (UniqueName: \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\") pod \"e167265d-4b6a-436a-8514-ab795d341fde\" (UID: \"e167265d-4b6a-436a-8514-ab795d341fde\") " Oct 30 20:00:20.539164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539001 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume" pod "07f74d5e-eeab-410d-8a8e-0dadf151c738" (UID: "07f74d5e-eeab-410d-8a8e-0dadf151c738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.539867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539844 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kc6\" (UniqueName: \"kubernetes.io/projected/96832826-53f3-4b64-b4aa-5eb28e9dc279-kube-api-access-x4kc6\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.539867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539866 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvwx6\" (UniqueName: \"kubernetes.io/projected/4b9701ae-3658-4c5e-bb57-9d604643d65f-kube-api-access-tvwx6\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.540038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539875 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e167265d-4b6a-436a-8514-ab795d341fde-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.540038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539885 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c72q\" (UniqueName: \"kubernetes.io/projected/638a7e34-94d9-4209-8562-2fb90d848609-kube-api-access-9c72q\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.540038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.539894 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f74d5e-eeab-410d-8a8e-0dadf151c738-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.541274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.541250 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz" pod "e167265d-4b6a-436a-8514-ab795d341fde" (UID: "e167265d-4b6a-436a-8514-ab795d341fde"). InnerVolumeSpecName "kube-api-access-bjbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.541379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.541332 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp" pod "07f74d5e-eeab-410d-8a8e-0dadf151c738" (UID: "07f74d5e-eeab-410d-8a8e-0dadf151c738"). InnerVolumeSpecName "kube-api-access-dh2kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.641535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.641465 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hv7\" (UniqueName: \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\") pod \"test-curl-7053-4925e2\" (UID: \"58c3c3d1-e108-4bce-82c9-1e7b1d87f232\") " pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:20.641934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.641913 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2kp\" (UniqueName: \"kubernetes.io/projected/07f74d5e-eeab-410d-8a8e-0dadf151c738-kube-api-access-dh2kp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.642000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.641938 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbsz\" (UniqueName: \"kubernetes.io/projected/e167265d-4b6a-436a-8514-ab795d341fde-kube-api-access-bjbsz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:20.743860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.743823 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hv7\" (UniqueName: \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\") pod \"test-curl-7053-4925e2\" (UID: \"58c3c3d1-e108-4bce-82c9-1e7b1d87f232\") " pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:20.764766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.764703 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hv7\" (UniqueName: \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\") pod \"test-curl-7053-4925e2\" (UID: \"58c3c3d1-e108-4bce-82c9-1e7b1d87f232\") " pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:20.770666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.770640 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" event={"ID":"13ad6203-4582-432c-9ef0-b8efab87118e","Type":"ContainerDied","Data":"76a53b19558132cccb7177c77b6dc1080b24110cde60e5c1cefc7b6a1b6ac5c6"} Oct 30 20:00:20.770792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.770651 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj" Oct 30 20:00:20.770792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.770701 4959 scope.go:122] "RemoveContainer" containerID="cdaba02e7b5bac5cf742ef4f424f2396cbc6e882be27f43af801174472b9d890" Oct 30 20:00:20.772479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.772452 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" Oct 30 20:00:20.772611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.772577 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2" event={"ID":"e167265d-4b6a-436a-8514-ab795d341fde","Type":"ContainerDied","Data":"727b418be8c0c5645db40cf3e98155344867aecbe374a5b3c30cd16068857597"} Oct 30 20:00:20.774186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.774150 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" Oct 30 20:00:20.774294 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.774178 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj" event={"ID":"4b9701ae-3658-4c5e-bb57-9d604643d65f","Type":"ContainerDied","Data":"8d5a81edd2292431e3f11c663919a22a50a8568c599def2fd8a3567ab5e911d6"} Oct 30 20:00:20.778053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.778025 4959 generic.go:356] "Generic (PLEG): container finished" podID="07f74d5e-eeab-410d-8a8e-0dadf151c738" containerID="24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0" exitCode=0 Oct 30 20:00:20.778157 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.778056 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" Oct 30 20:00:20.778157 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.778125 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" event={"ID":"07f74d5e-eeab-410d-8a8e-0dadf151c738","Type":"ContainerDied","Data":"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0"} Oct 30 20:00:20.778261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.778165 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7" event={"ID":"07f74d5e-eeab-410d-8a8e-0dadf151c738","Type":"ContainerDied","Data":"3f92f089ce73631e9edacb1f27b0905cb6ac9c74f20ea03b4b553c43c67e5f32"} Oct 30 20:00:20.781976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.781222 4959 generic.go:356] "Generic (PLEG): container finished" podID="96832826-53f3-4b64-b4aa-5eb28e9dc279" containerID="4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b" exitCode=0 Oct 30 20:00:20.781976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.781307 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" event={"ID":"96832826-53f3-4b64-b4aa-5eb28e9dc279","Type":"ContainerDied","Data":"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b"} Oct 30 20:00:20.781976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.781337 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" event={"ID":"96832826-53f3-4b64-b4aa-5eb28e9dc279","Type":"ContainerDied","Data":"8b48e4f1dfb7753021c41d94aa2e5f45354d8cc213da161ec0da21022fc13638"} Oct 30 20:00:20.781976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.781395 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb" Oct 30 20:00:20.783956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.783159 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" Oct 30 20:00:20.783956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.783176 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm" event={"ID":"638a7e34-94d9-4209-8562-2fb90d848609","Type":"ContainerDied","Data":"9bf2f85ed49b6e4cd3c66d9e2817b65544cba9e3dc79cf96da21b89cfd46a00c"} Oct 30 20:00:20.793630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.793608 4959 scope.go:122] "RemoveContainer" containerID="5544dedcae4d28a539ce1c24612ee1e9143b219809e496302857beb3f42e11c7" Oct 30 20:00:20.804715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.804694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.806422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.806406 4959 scope.go:122] "RemoveContainer" containerID="955bded5c3a5a62876ecf41dae144d784532828efb818e7fd6edcaee828f52db" Oct 30 20:00:20.813996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.813974 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:20.817428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.817401 4959 scope.go:122] "RemoveContainer" containerID="24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0" Oct 30 20:00:20.833320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.833304 4959 scope.go:122] "RemoveContainer" containerID="24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0" Oct 30 20:00:20.833729 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:20.833708 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0\": container with ID starting with 24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0 not found: ID does not exist" containerID="24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0" Oct 30 20:00:20.833789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.833736 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0"} err="failed to get container status \"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0\": rpc error: code = NotFound desc = could not find container \"24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0\": container with ID starting with 24d7aa4cff4de550278dcff913059f33b0727105302d4efe2cc26622ddf9e0b0 not found: ID does not exist" Oct 30 20:00:20.833789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.833754 4959 scope.go:122] "RemoveContainer" containerID="4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b" Oct 30 20:00:20.847845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.847780 4959 scope.go:122] "RemoveContainer" containerID="4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b" Oct 30 20:00:20.848480 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:20.848443 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b\": container with ID starting with 4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b not found: ID does not exist" containerID="4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b" Oct 30 20:00:20.848480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.848485 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b"} err="failed to get container status \"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b\": rpc error: code = NotFound desc = could not find container \"4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b\": container with ID starting with 4fdab101503421b6fb698473cc31a6430126bb1dbe5808b3cb3ba29a6804c08b not found: ID does not exist" Oct 30 20:00:20.848480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.848531 4959 scope.go:122] "RemoveContainer" containerID="369a6d017ef2ae7fbd5c204e044acf2116209c6fc3f7869b5fea709c832b2557" Oct 30 20:00:20.866888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.855273 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:20.926813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.899101 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889840-txnh2"] Oct 30 20:00:21.003276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:20.998291 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:21.005178 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.005114 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889840-vx6wj"] Oct 30 20:00:21.048577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.048459 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:21.059332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.059302 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a165e5-29889840-n9tqm"] Oct 30 20:00:21.096822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.096789 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 20:00:21.104009 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.103982 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-wgmg7"] Oct 30 20:00:21.135617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.135587 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:21.139717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.139696 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c8fa69-29889840-v62gj"] Oct 30 20:00:21.179916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.179888 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 20:00:21.198584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.198560 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-vqrqb"] Oct 30 20:00:21.243613 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:21.243586 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58c3c3d1_e108_4bce_82c9_1e7b1d87f232.slice/crio-dc0888c2140629626f81076198e86b26158acad09e5f42697251576a2668de37 WatchSource:0}: Error finding container dc0888c2140629626f81076198e86b26158acad09e5f42697251576a2668de37: Status 404 returned error can't find the container with id dc0888c2140629626f81076198e86b26158acad09e5f42697251576a2668de37 Oct 30 20:00:21.244577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.244551 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-4925e2"] Oct 30 20:00:21.797080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.797044 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-4925e2" event={"ID":"58c3c3d1-e108-4bce-82c9-1e7b1d87f232","Type":"ContainerStarted","Data":"e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa"} Oct 30 20:00:21.797080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.797085 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-4925e2" event={"ID":"58c3c3d1-e108-4bce-82c9-1e7b1d87f232","Type":"ContainerStarted","Data":"dc0888c2140629626f81076198e86b26158acad09e5f42697251576a2668de37"} Oct 30 20:00:21.839478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:21.839429 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-4925e2" podStartSLOduration=1.839414726 podStartE2EDuration="1.839414726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:21.836342602 +0000 UTC m=+1545.965576523" watchObservedRunningTime="2026-10-30 20:00:21.839414726 +0000 UTC m=+1545.968648647" Oct 30 20:00:22.002442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.002404 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f74d5e-eeab-410d-8a8e-0dadf151c738" path="/var/lib/kubelet/pods/07f74d5e-eeab-410d-8a8e-0dadf151c738/volumes" Oct 30 20:00:22.003040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.003020 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13ad6203-4582-432c-9ef0-b8efab87118e" path="/var/lib/kubelet/pods/13ad6203-4582-432c-9ef0-b8efab87118e/volumes" Oct 30 20:00:22.003440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.003426 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9701ae-3658-4c5e-bb57-9d604643d65f" path="/var/lib/kubelet/pods/4b9701ae-3658-4c5e-bb57-9d604643d65f/volumes" Oct 30 20:00:22.004108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.004088 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638a7e34-94d9-4209-8562-2fb90d848609" path="/var/lib/kubelet/pods/638a7e34-94d9-4209-8562-2fb90d848609/volumes" Oct 30 20:00:22.004548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.004531 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96832826-53f3-4b64-b4aa-5eb28e9dc279" path="/var/lib/kubelet/pods/96832826-53f3-4b64-b4aa-5eb28e9dc279/volumes" Oct 30 20:00:22.004940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.004927 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e167265d-4b6a-436a-8514-ab795d341fde" path="/var/lib/kubelet/pods/e167265d-4b6a-436a-8514-ab795d341fde/volumes" Oct 30 20:00:22.005658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.005643 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" path="/var/lib/kubelet/pods/fc23b796-bbd4-4b71-bee4-66e7cea06c97/volumes" Oct 30 20:00:22.632152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.632119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:22.752581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.746131 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-4925e2"] Oct 30 20:00:22.786737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.786416 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-4925e2"] Oct 30 20:00:22.835295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.835264 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-9d87c2"] Oct 30 20:00:22.836594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836569 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:22.836594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836595 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc23b796-bbd4-4b71-bee4-66e7cea06c97" containerName="cleanup" Oct 30 20:00:22.836792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836646 4959 cpu_manager.go:414] "containerMap: removing container" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" containerName="test-curl-7053-4925e2" Oct 30 20:00:22.836792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836655 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" containerName="test-curl-7053-4925e2" Oct 30 20:00:22.836927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836912 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="638a7e34-94d9-4209-8562-2fb90d848609" containerName="cleanup" Oct 30 20:00:22.836984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.836941 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" containerName="test-curl-7053-4925e2" Oct 30 20:00:22.843216 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.843195 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:22.856298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.856274 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-9d87c2"] Oct 30 20:00:22.973351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:22.973318 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22k79\" (UniqueName: \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\") pod \"test-curl-7053-9d87c2\" (UID: \"a377004a-fb2e-485e-99e0-42b008e93624\") " pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:23.074510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.074462 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22k79\" (UniqueName: \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\") pod \"test-curl-7053-9d87c2\" (UID: \"a377004a-fb2e-485e-99e0-42b008e93624\") " pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:23.096351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.096316 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22k79\" (UniqueName: \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\") pod \"test-curl-7053-9d87c2\" (UID: \"a377004a-fb2e-485e-99e0-42b008e93624\") " pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:23.165637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.165608 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:23.473921 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:23.473889 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda377004a_fb2e_485e_99e0_42b008e93624.slice/crio-af2c1073548466f90f90ffae93861e41ad6e60b56c6ad8eb376fba3fb365ab92 WatchSource:0}: Error finding container af2c1073548466f90f90ffae93861e41ad6e60b56c6ad8eb376fba3fb365ab92: Status 404 returned error can't find the container with id af2c1073548466f90f90ffae93861e41ad6e60b56c6ad8eb376fba3fb365ab92 Oct 30 20:00:23.475605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.475581 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-9d87c2"] Oct 30 20:00:23.820288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.820182 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-9d87c2" event={"ID":"a377004a-fb2e-485e-99e0-42b008e93624","Type":"ContainerStarted","Data":"35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99"} Oct 30 20:00:23.820288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.820227 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-9d87c2" event={"ID":"a377004a-fb2e-485e-99e0-42b008e93624","Type":"ContainerStarted","Data":"af2c1073548466f90f90ffae93861e41ad6e60b56c6ad8eb376fba3fb365ab92"} Oct 30 20:00:23.820555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.820426 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-4925e2" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" containerName="test-curl-7053-4925e2" containerID="cri-o://e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa" gracePeriod=2 Oct 30 20:00:23.862897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:23.862848 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-9d87c2" podStartSLOduration=1.86283375 podStartE2EDuration="1.86283375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:23.859986332 +0000 UTC m=+1547.989220253" watchObservedRunningTime="2026-10-30 20:00:23.86283375 +0000 UTC m=+1547.992067671" Oct 30 20:00:23.874846 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:23.874806 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-4925e2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:24.179117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.179078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:24.742655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.742613 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-9d87c2"] Oct 30 20:00:24.748326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.748295 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-9d87c2"] Oct 30 20:00:24.901025 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.900997 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-59d4a1"] Oct 30 20:00:24.901846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.901830 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a377004a-fb2e-485e-99e0-42b008e93624" containerName="test-curl-7053-9d87c2" Oct 30 20:00:24.901896 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.901847 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a377004a-fb2e-485e-99e0-42b008e93624" containerName="test-curl-7053-9d87c2" Oct 30 20:00:24.901983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.901974 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a377004a-fb2e-485e-99e0-42b008e93624" containerName="test-curl-7053-9d87c2" Oct 30 20:00:24.905848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.905832 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:24.921856 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:24.921835 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-59d4a1"] Oct 30 20:00:25.043903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.043828 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657wn\" (UniqueName: \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\") pod \"test-curl-7053-59d4a1\" (UID: \"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5\") " pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:25.145271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.145240 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657wn\" (UniqueName: \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\") pod \"test-curl-7053-59d4a1\" (UID: \"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5\") " pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:25.165430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.165404 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657wn\" (UniqueName: \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\") pod \"test-curl-7053-59d4a1\" (UID: \"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5\") " pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:25.219041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.219012 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:25.529131 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:25.529106 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b6339d7_fbe6_472c_81c6_bb0f6918d1d5.slice/crio-7b4462556ecebcb6aacdd3132d7d60cd4a1bb9772804a80a99b7658e9f81ec27 WatchSource:0}: Error finding container 7b4462556ecebcb6aacdd3132d7d60cd4a1bb9772804a80a99b7658e9f81ec27: Status 404 returned error can't find the container with id 7b4462556ecebcb6aacdd3132d7d60cd4a1bb9772804a80a99b7658e9f81ec27 Oct 30 20:00:25.530991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.530971 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-59d4a1"] Oct 30 20:00:25.923266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.923239 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-59d4a1" event={"ID":"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5","Type":"ContainerStarted","Data":"7b4462556ecebcb6aacdd3132d7d60cd4a1bb9772804a80a99b7658e9f81ec27"} Oct 30 20:00:25.924641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:25.923438 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-9d87c2" podUID="a377004a-fb2e-485e-99e0-42b008e93624" containerName="test-curl-7053-9d87c2" containerID="cri-o://35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99" gracePeriod=2 Oct 30 20:00:25.937938 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:25.937911 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-9d87c2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="a377004a-fb2e-485e-99e0-42b008e93624" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:26.010837 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:26.010808 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-9d87c2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="a377004a-fb2e-485e-99e0-42b008e93624" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:26.090956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.090896 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:26.105442 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:26.105415 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-4925e2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:26.182898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.182875 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.203261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.203231 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\" (UniqueName: \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\") pod \"58c3c3d1-e108-4bce-82c9-1e7b1d87f232\" (UID: \"58c3c3d1-e108-4bce-82c9-1e7b1d87f232\") " Oct 30 20:00:26.205438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.205408 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7" pod "58c3c3d1-e108-4bce-82c9-1e7b1d87f232" (UID: "58c3c3d1-e108-4bce-82c9-1e7b1d87f232"). InnerVolumeSpecName "kube-api-access-q5hv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:26.306125 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.306077 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hv7\" (UniqueName: \"kubernetes.io/projected/58c3c3d1-e108-4bce-82c9-1e7b1d87f232-kube-api-access-q5hv7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:26.934575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.934542 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-59d4a1" event={"ID":"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5","Type":"ContainerStarted","Data":"39493d0674ce2f67584d7a8a089a9f3a5ae17fa7881bf1fa4a2bfe43f76f84b1"} Oct 30 20:00:26.936544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.936519 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:26.936544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.936538 4959 scope.go:122] "RemoveContainer" containerID="e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa" Oct 30 20:00:26.936810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.936476 4959 generic.go:356] "Generic (PLEG): container finished" podID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" containerID="e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa" exitCode=137 Oct 30 20:00:26.947851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.947835 4959 scope.go:122] "RemoveContainer" containerID="e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa" Oct 30 20:00:26.948139 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:26.948115 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa\": container with ID starting with e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa not found: ID does not exist" containerID="e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa" Oct 30 20:00:26.948201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.948152 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa"} err="failed to get container status \"e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa\": rpc error: code = NotFound desc = could not find container \"e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa\": container with ID starting with e96b6deddcb36810f97af212d05a9e66db1b19c6c1c5236444dc9fb522b250aa not found: ID does not exist" Oct 30 20:00:26.977030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:26.976982 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-59d4a1" podStartSLOduration=2.976968924 podStartE2EDuration="2.976968924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:26.975014598 +0000 UTC m=+1551.104248526" watchObservedRunningTime="2026-10-30 20:00:26.976968924 +0000 UTC m=+1551.106202845" Oct 30 20:00:26.989429 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:26.989402 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-4925e2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:27.003869 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:27.003840 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-4925e2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" pod="opendatahub/test-curl-7053-4925e2" Oct 30 20:00:27.028095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:27.028072 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:27.846711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:27.846679 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-59d4a1"] Oct 30 20:00:27.857883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:27.857858 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-59d4a1"] Oct 30 20:00:28.003886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.003001 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c3c3d1-e108-4bce-82c9-1e7b1d87f232" path="/var/lib/kubelet/pods/58c3c3d1-e108-4bce-82c9-1e7b1d87f232/volumes" Oct 30 20:00:28.017712 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.017688 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-7ab1f8"] Oct 30 20:00:28.018606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.018590 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" containerName="test-curl-7053-59d4a1" Oct 30 20:00:28.018648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.018608 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" containerName="test-curl-7053-59d4a1" Oct 30 20:00:28.018763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.018754 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" containerName="test-curl-7053-59d4a1" Oct 30 20:00:28.022559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.022543 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:28.051562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.051540 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-7ab1f8"] Oct 30 20:00:28.136715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.136378 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8z5\" (UniqueName: \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\") pod \"test-curl-7053-7ab1f8\" (UID: \"f8fb65ce-c991-40f1-8c3a-85a224ad1837\") " pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:28.172686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.172657 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:28.236741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.236709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.237629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.237609 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8z5\" (UniqueName: \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\") pod \"test-curl-7053-7ab1f8\" (UID: \"f8fb65ce-c991-40f1-8c3a-85a224ad1837\") " pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:28.258413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.258390 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8z5\" (UniqueName: \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\") pod \"test-curl-7053-7ab1f8\" (UID: \"f8fb65ce-c991-40f1-8c3a-85a224ad1837\") " pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:28.335721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.335692 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:28.361518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.356301 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\" (UniqueName: \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\") pod \"a377004a-fb2e-485e-99e0-42b008e93624\" (UID: \"a377004a-fb2e-485e-99e0-42b008e93624\") " Oct 30 20:00:28.374865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.372737 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79" pod "a377004a-fb2e-485e-99e0-42b008e93624" (UID: "a377004a-fb2e-485e-99e0-42b008e93624"). InnerVolumeSpecName "kube-api-access-22k79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:28.462517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.460122 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22k79\" (UniqueName: \"kubernetes.io/projected/a377004a-fb2e-485e-99e0-42b008e93624-kube-api-access-22k79\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:28.766966 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:28.766937 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8fb65ce_c991_40f1_8c3a_85a224ad1837.slice/crio-46b1008f8d6efd4b5ba1b10f679c8113ea19aece51ca406dfba98af6b03ba16c WatchSource:0}: Error finding container 46b1008f8d6efd4b5ba1b10f679c8113ea19aece51ca406dfba98af6b03ba16c: Status 404 returned error can't find the container with id 46b1008f8d6efd4b5ba1b10f679c8113ea19aece51ca406dfba98af6b03ba16c Oct 30 20:00:28.767939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.767916 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-7ab1f8"] Oct 30 20:00:28.922507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.922448 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:00:28.928781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.928736 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:28.946887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.946855 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:00:28.956233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.956201 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0506c9\"/\"kube-root-ca.crt\"" Oct 30 20:00:28.956371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.956344 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0506c9\"/\"openshift-service-ca.crt\"" Oct 30 20:00:28.960237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.960127 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0506c9\"/\"test-model-0506c9-kserve-self-signed-certs\"" Oct 30 20:00:28.960237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.960204 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0506c9\"/\"default-dockercfg-qkjc8\"" Oct 30 20:00:28.964860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.964830 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-7ab1f8" event={"ID":"f8fb65ce-c991-40f1-8c3a-85a224ad1837","Type":"ContainerStarted","Data":"813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d"} Oct 30 20:00:28.965083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.964876 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-7ab1f8" event={"ID":"f8fb65ce-c991-40f1-8c3a-85a224ad1837","Type":"ContainerStarted","Data":"46b1008f8d6efd4b5ba1b10f679c8113ea19aece51ca406dfba98af6b03ba16c"} Oct 30 20:00:28.969568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.969541 4959 generic.go:356] "Generic (PLEG): container finished" podID="a377004a-fb2e-485e-99e0-42b008e93624" containerID="35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99" exitCode=137 Oct 30 20:00:28.969678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.969589 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:28.969678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.969650 4959 scope.go:122] "RemoveContainer" containerID="35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99" Oct 30 20:00:28.969829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.969803 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-59d4a1" podUID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" containerName="test-curl-7053-59d4a1" containerID="cri-o://39493d0674ce2f67584d7a8a089a9f3a5ae17fa7881bf1fa4a2bfe43f76f84b1" gracePeriod=2 Oct 30 20:00:28.975329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.975274 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.986670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.986650 4959 scope.go:122] "RemoveContainer" containerID="35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99" Oct 30 20:00:28.987433 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:28.987405 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99\": container with ID starting with 35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99 not found: ID does not exist" containerID="35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99" Oct 30 20:00:28.987859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:28.987444 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99"} err="failed to get container status \"35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99\": rpc error: code = NotFound desc = could not find container \"35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99\": container with ID starting with 35d9c0421579cff85dcf2af2d2e74fe52762068174bbc0821a681c07da9ccd99 not found: ID does not exist" Oct 30 20:00:29.045644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.045551 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-7ab1f8" podStartSLOduration=2.045536937 podStartE2EDuration="2.045536937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:29.042220477 +0000 UTC m=+1553.171454399" watchObservedRunningTime="2026-10-30 20:00:29.045536937 +0000 UTC m=+1553.174770857" Oct 30 20:00:29.057455 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:29.057427 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-9d87c2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="a377004a-fb2e-485e-99e0-42b008e93624" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:29.069547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.069516 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.069731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.069710 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.069780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.069740 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6j8\" (UniqueName: \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.069885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.069867 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.069976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.069962 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.070093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.070078 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.197390 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.197601 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.197632 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6j8\" (UniqueName: \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.197845 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.197932 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.200526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.198049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.207594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.202272 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.207594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.202898 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.223705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.223222 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.232432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.225730 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.253155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.235929 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.280557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.273672 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6j8\" (UniqueName: \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\") pod \"test-model-0506c9-kserve-7b6d585b9c-rgrpg\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.281133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.281107 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:29.750618 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:29.750573 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92972dd3_69d1_4316_afe5_6a58cd965d96.slice/crio-16ca605bbed24c93f7dfe34d692b1d7899801729f0fdb39795a6659a29ec2ba0 WatchSource:0}: Error finding container 16ca605bbed24c93f7dfe34d692b1d7899801729f0fdb39795a6659a29ec2ba0: Status 404 returned error can't find the container with id 16ca605bbed24c93f7dfe34d692b1d7899801729f0fdb39795a6659a29ec2ba0 Oct 30 20:00:29.752123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.752101 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:00:29.978945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.978915 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" event={"ID":"92972dd3-69d1-4316-afe5-6a58cd965d96","Type":"ContainerStarted","Data":"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c"} Oct 30 20:00:29.978945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:29.978955 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" event={"ID":"92972dd3-69d1-4316-afe5-6a58cd965d96","Type":"ContainerStarted","Data":"16ca605bbed24c93f7dfe34d692b1d7899801729f0fdb39795a6659a29ec2ba0"} Oct 30 20:00:30.002949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.002882 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a377004a-fb2e-485e-99e0-42b008e93624" path="/var/lib/kubelet/pods/a377004a-fb2e-485e-99e0-42b008e93624/volumes" Oct 30 20:00:30.045839 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:30.045809 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-9d87c2\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="a377004a-fb2e-485e-99e0-42b008e93624" pod="opendatahub/test-curl-7053-9d87c2" Oct 30 20:00:30.049955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.049750 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" podStartSLOduration=2.049736644 podStartE2EDuration="2.049736644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:30.031040393 +0000 UTC m=+1554.160274314" watchObservedRunningTime="2026-10-30 20:00:30.049736644 +0000 UTC m=+1554.178970564" Oct 30 20:00:30.051391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.051318 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-7ab1f8"] Oct 30 20:00:30.057781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.057755 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-7ab1f8"] Oct 30 20:00:30.075312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.075280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.256583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.256452 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.284264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.284237 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:30.363969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.363936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.586559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:30.586458 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.019215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.018939 4959 generic.go:356] "Generic (PLEG): container finished" podID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" containerID="39493d0674ce2f67584d7a8a089a9f3a5ae17fa7881bf1fa4a2bfe43f76f84b1" exitCode=137 Oct 30 20:00:31.021474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.020205 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-7ab1f8" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" containerName="test-curl-7053-7ab1f8" containerID="cri-o://813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d" gracePeriod=2 Oct 30 20:00:31.035414 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:31.035367 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-7ab1f8\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:31.065531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.065448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.255578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.255274 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.463391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.462273 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:31.486305 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:31.484086 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-7ab1f8\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:31.564568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.563644 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\" (UniqueName: \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\") pod \"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5\" (UID: \"8b6339d7-fbe6-472c-81c6-bb0f6918d1d5\") " Oct 30 20:00:31.609266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.600710 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn" pod "8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" (UID: "8b6339d7-fbe6-472c-81c6-bb0f6918d1d5"). InnerVolumeSpecName "kube-api-access-657wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:31.666740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.666705 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657wn\" (UniqueName: \"kubernetes.io/projected/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5-kube-api-access-657wn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:31.998552 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:31.998521 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6339d7-fbe6-472c-81c6-bb0f6918d1d5" path="/var/lib/kubelet/pods/8b6339d7-fbe6-472c-81c6-bb0f6918d1d5/volumes" Oct 30 20:00:32.029552 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:32.029526 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-59d4a1" Oct 30 20:00:32.029717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:32.029532 4959 scope.go:122] "RemoveContainer" containerID="39493d0674ce2f67584d7a8a089a9f3a5ae17fa7881bf1fa4a2bfe43f76f84b1" Oct 30 20:00:32.204968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:32.204831 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.250468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:33.250435 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:33.321030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:33.320943 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\" (UniqueName: \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\") pod \"f8fb65ce-c991-40f1-8c3a-85a224ad1837\" (UID: \"f8fb65ce-c991-40f1-8c3a-85a224ad1837\") " Oct 30 20:00:33.323331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:33.323305 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5" pod "f8fb65ce-c991-40f1-8c3a-85a224ad1837" (UID: "f8fb65ce-c991-40f1-8c3a-85a224ad1837"). InnerVolumeSpecName "kube-api-access-2z8z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:33.423598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:33.423563 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z8z5\" (UniqueName: \"kubernetes.io/projected/f8fb65ce-c991-40f1-8c3a-85a224ad1837-kube-api-access-2z8z5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:33.710061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:33.710023 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:34.003071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.002984 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" path="/var/lib/kubelet/pods/f8fb65ce-c991-40f1-8c3a-85a224ad1837/volumes" Oct 30 20:00:34.050719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.050684 4959 generic.go:356] "Generic (PLEG): container finished" podID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" containerID="813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d" exitCode=137 Oct 30 20:00:34.050899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.050729 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-7ab1f8" Oct 30 20:00:34.050899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.050783 4959 scope.go:122] "RemoveContainer" containerID="813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d" Oct 30 20:00:34.061266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.061092 4959 scope.go:122] "RemoveContainer" containerID="813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d" Oct 30 20:00:34.061385 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:34.061366 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d\": container with ID starting with 813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d not found: ID does not exist" containerID="813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d" Oct 30 20:00:34.061433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.061393 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d"} err="failed to get container status \"813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d\": rpc error: code = NotFound desc = could not find container \"813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d\": container with ID starting with 813ce02011a364d74bdc17ed0c56ed17bb16cde722eee6f58eee8aaeff90223d not found: ID does not exist" Oct 30 20:00:34.394234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:34.394200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.024258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:35.024226 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.133161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:35.133128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.500338 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:35.500306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.541702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:36.541669 4959 scope.go:122] "RemoveContainer" containerID="a38c80072eb4bb2c0cefa8709fdef0ef43819abee8093c7f61379f52126ed98f" Oct 30 20:00:36.679341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:36.679295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.798048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:36.797948 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:37.162558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:37.156742 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:37.662319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:37.662290 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.284291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:39.284258 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:39.297962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:39.297934 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:39.368729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:39.368697 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.169335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:40.169304 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:00:40.982145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:40.982106 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.526989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:42.526947 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 20:00:42.527532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:42.527225 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" podUID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" containerName="maas-api" containerID="cri-o://d3e23bc25378852f81c232333e8a5f6bacad0989fc1e60fe3f37234a9b11f3e0" gracePeriod=30 Oct 30 20:00:42.549716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:42.549684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.259691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.259660 4959 generic.go:356] "Generic (PLEG): container finished" podID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" containerID="d3e23bc25378852f81c232333e8a5f6bacad0989fc1e60fe3f37234a9b11f3e0" exitCode=0 Oct 30 20:00:43.259892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.259874 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" event={"ID":"0fb1315b-3516-43ed-aeca-a9f0d7587c0e","Type":"ContainerDied","Data":"d3e23bc25378852f81c232333e8a5f6bacad0989fc1e60fe3f37234a9b11f3e0"} Oct 30 20:00:43.288844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.272233 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 20:00:43.353368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.353003 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\") pod \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " Oct 30 20:00:43.353368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.353082 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\") pod \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " Oct 30 20:00:43.353368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.353275 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\" (UniqueName: \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\") pod \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " Oct 30 20:00:43.397588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.386714 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca" pod "0fb1315b-3516-43ed-aeca-a9f0d7587c0e" (UID: "0fb1315b-3516-43ed-aeca-a9f0d7587c0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:43.422577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.414915 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\") pod \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\" (UID: \"0fb1315b-3516-43ed-aeca-a9f0d7587c0e\") " Oct 30 20:00:43.431770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.431708 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.485597 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.471668 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs" pod "0fb1315b-3516-43ed-aeca-a9f0d7587c0e" (UID: "0fb1315b-3516-43ed-aeca-a9f0d7587c0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:43.533517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.532680 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj" pod "0fb1315b-3516-43ed-aeca-a9f0d7587c0e" (UID: "0fb1315b-3516-43ed-aeca-a9f0d7587c0e"). InnerVolumeSpecName "kube-api-access-v2jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:43.533989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.533635 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls" pod "0fb1315b-3516-43ed-aeca-a9f0d7587c0e" (UID: "0fb1315b-3516-43ed-aeca-a9f0d7587c0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:43.552727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.534442 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jdj\" (UniqueName: \"kubernetes.io/projected/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-kube-api-access-v2jdj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.552727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.534475 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.552727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.548464 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.584942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.559604 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb1315b-3516-43ed-aeca-a9f0d7587c0e-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:00:43.762581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.750367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.865215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:43.865186 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.249905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.249866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.268640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.268616 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" Oct 30 20:00:44.268837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.268657 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt" event={"ID":"0fb1315b-3516-43ed-aeca-a9f0d7587c0e","Type":"ContainerDied","Data":"6ced13c41f9cad8fc7f8ff16ecc5573740e31ed6322c210a575a870181de7af1"} Oct 30 20:00:44.268837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.268712 4959 scope.go:122] "RemoveContainer" containerID="d3e23bc25378852f81c232333e8a5f6bacad0989fc1e60fe3f37234a9b11f3e0" Oct 30 20:00:44.328963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.328928 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 20:00:44.342165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:44.342088 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d97a5-674fbbfc75-gb5qt"] Oct 30 20:00:45.595381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:45.595331 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.999924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:45.999892 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" path="/var/lib/kubelet/pods/0fb1315b-3516-43ed-aeca-a9f0d7587c0e/volumes" Oct 30 20:00:46.067951 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:00:46.067912 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:00:46.510974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:46.510945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.089049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:47.089016 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.880077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:47.880034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:47.997785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:47.997755 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.101397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:48.101365 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.537290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:48.537254 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.719617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:49.719583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.033703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:51.033663 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.163307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:51.152028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.315480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:51.315378 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.468216 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:51.468180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.058636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:53.058600 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.900614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:54.900563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.636248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:55.636214 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.752537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:55.747837 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.023442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:56.023337 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.087357 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:57.087314 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.208638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:57.208600 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.492602 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:57.492567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:58.917801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.917773 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:00:58.918932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.918898 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" containerName="maas-api" Oct 30 20:00:58.918932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.918921 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" containerName="maas-api" Oct 30 20:00:58.919131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.918974 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" containerName="test-curl-7053-7ab1f8" Oct 30 20:00:58.919131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.918981 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" containerName="test-curl-7053-7ab1f8" Oct 30 20:00:58.919249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.919176 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8fb65ce-c991-40f1-8c3a-85a224ad1837" containerName="test-curl-7053-7ab1f8" Oct 30 20:00:58.919249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.919197 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb1315b-3516-43ed-aeca-a9f0d7587c0e" containerName="maas-api" Oct 30 20:00:58.923847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.923818 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:58.949845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.949823 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:00:58.957969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.957937 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb16ce\"/\"test-model-eb16ce-kserve-self-signed-certs\"" Oct 30 20:00:58.958092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.958012 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb16ce\"/\"kube-root-ca.crt\"" Oct 30 20:00:58.962141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.962121 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb16ce\"/\"openshift-service-ca.crt\"" Oct 30 20:00:58.962251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.962159 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb16ce\"/\"default-dockercfg-v2bbf\"" Oct 30 20:00:58.974465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:58.974421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.064977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.064948 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.065173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.065022 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.065248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.065199 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.065380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.065291 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.065474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.065413 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.065474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.065455 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxn9v\" (UniqueName: \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180071 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180303 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180368 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180457 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.180536 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxn9v\" (UniqueName: \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.186595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.181353 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.207517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.194778 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.207517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.206113 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.212518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.209067 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.248568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.248204 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.248875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.248852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxn9v\" (UniqueName: \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\") pod \"test-model-eb16ce-kserve-5b486c69c9-5hcps\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.263354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.263322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.540518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.538159 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:00:59.920379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:00:59.919936 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:00:59.941518 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:00:59.933614 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d04c0c3_68a1_4e3c_b04f_7795249608fd.slice/crio-4e1847c93732d4edd95c7c4e49cd2dea8125529fc9aa86162dc4d3309d679023 WatchSource:0}: Error finding container 4e1847c93732d4edd95c7c4e49cd2dea8125529fc9aa86162dc4d3309d679023: Status 404 returned error can't find the container with id 4e1847c93732d4edd95c7c4e49cd2dea8125529fc9aa86162dc4d3309d679023 Oct 30 20:01:00.549285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:00.549246 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" event={"ID":"6d04c0c3-68a1-4e3c-b04f-7795249608fd","Type":"ContainerStarted","Data":"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1"} Oct 30 20:01:00.549458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:00.549294 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" event={"ID":"6d04c0c3-68a1-4e3c-b04f-7795249608fd","Type":"ContainerStarted","Data":"4e1847c93732d4edd95c7c4e49cd2dea8125529fc9aa86162dc4d3309d679023"} Oct 30 20:01:00.594262 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:00.594205 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" podStartSLOduration=2.594189439 podStartE2EDuration="2.594189439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:00.590392301 +0000 UTC m=+1584.719626224" watchObservedRunningTime="2026-10-30 20:01:00.594189439 +0000 UTC m=+1584.723423362" Oct 30 20:01:01.134663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:01.134631 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.539484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:01.539412 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:01:02.128288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:02.128254 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 20:01:02.269658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:02.269624 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.424664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:02.424629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.709204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:02.709123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.258399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:04.258357 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.811846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:05.811810 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.223556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:06.223523 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.334120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:06.334072 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.893046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:06.893014 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.011562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:07.011511 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.128611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:07.128559 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" podUID="473aba10-bccb-47b9-b403-7fb253d16386" containerName="payload-processing" containerID="cri-o://8583aa8a91bf97bb8b311b2559df04419a61ae4c9ac09d11785eea05ef6a06b9" gracePeriod=25 Oct 30 20:01:07.239150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:07.239106 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 20:01:07.239353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:07.239330 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerName="maas-api" containerID="cri-o://d172ff11140a0f267f5da49afbc55e52d079b908b07a5cad59f2b3541e4a6ab3" gracePeriod=30 Oct 30 20:01:07.303139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:07.303103 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.304923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.304880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.307224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.307194 4959 generic.go:356] "Generic (PLEG): container finished" podID="473aba10-bccb-47b9-b403-7fb253d16386" containerID="8583aa8a91bf97bb8b311b2559df04419a61ae4c9ac09d11785eea05ef6a06b9" exitCode=0 Oct 30 20:01:08.307390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.307372 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" event={"ID":"473aba10-bccb-47b9-b403-7fb253d16386","Type":"ContainerDied","Data":"8583aa8a91bf97bb8b311b2559df04419a61ae4c9ac09d11785eea05ef6a06b9"} Oct 30 20:01:08.310099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.310061 4959 generic.go:356] "Generic (PLEG): container finished" podID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerID="d172ff11140a0f267f5da49afbc55e52d079b908b07a5cad59f2b3541e4a6ab3" exitCode=0 Oct 30 20:01:08.310236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.310210 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" event={"ID":"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984","Type":"ContainerDied","Data":"d172ff11140a0f267f5da49afbc55e52d079b908b07a5cad59f2b3541e4a6ab3"} Oct 30 20:01:08.321242 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.321221 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 20:01:08.325360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.325342 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 20:01:08.414630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.414592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.490770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.490681 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") pod \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " Oct 30 20:01:08.490944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.490816 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\") pod \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " Oct 30 20:01:08.490944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.490927 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\") pod \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " Oct 30 20:01:08.491080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.491063 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\") pod \"473aba10-bccb-47b9-b403-7fb253d16386\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " Oct 30 20:01:08.491144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.491121 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\" (UniqueName: \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\") pod \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\" (UID: \"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984\") " Oct 30 20:01:08.491226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.491211 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\" (UniqueName: \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\") pod \"473aba10-bccb-47b9-b403-7fb253d16386\" (UID: \"473aba10-bccb-47b9-b403-7fb253d16386\") " Oct 30 20:01:08.491345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.491317 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca" pod "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" (UID: "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:08.491411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.491378 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume" pod "473aba10-bccb-47b9-b403-7fb253d16386" (UID: "473aba10-bccb-47b9-b403-7fb253d16386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:08.492737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.492715 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.492737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.492739 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473aba10-bccb-47b9-b403-7fb253d16386-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.493605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.493584 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls" pod "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" (UID: "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:08.493701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.493609 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs" pod "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" (UID: "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:08.493836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.493818 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv" pod "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" (UID: "2cd1615f-32ce-49d5-ae8e-77ceeb1e5984"). InnerVolumeSpecName "kube-api-access-rnwbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:08.493900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.493844 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn" pod "473aba10-bccb-47b9-b403-7fb253d16386" (UID: "473aba10-bccb-47b9-b403-7fb253d16386"). InnerVolumeSpecName "kube-api-access-9qbkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:08.599520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.597488 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.599520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.597545 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.599520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.597559 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwbv\" (UniqueName: \"kubernetes.io/projected/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984-kube-api-access-rnwbv\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.599520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.597574 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbkn\" (UniqueName: \"kubernetes.io/projected/473aba10-bccb-47b9-b403-7fb253d16386-kube-api-access-9qbkn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:08.651152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:08.651113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.319768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.319740 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" Oct 30 20:01:09.319768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.319761 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" event={"ID":"2cd1615f-32ce-49d5-ae8e-77ceeb1e5984","Type":"ContainerDied","Data":"a8a26ded63003ee13f573a20eb3aa4ded48625989766b52b2a69b657f001f5b5"} Oct 30 20:01:09.320375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.319828 4959 scope.go:122] "RemoveContainer" containerID="d172ff11140a0f267f5da49afbc55e52d079b908b07a5cad59f2b3541e4a6ab3" Oct 30 20:01:09.321888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.321862 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" Oct 30 20:01:09.322026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.321870 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm" event={"ID":"473aba10-bccb-47b9-b403-7fb253d16386","Type":"ContainerDied","Data":"605a46cd52be23aaf2fc487690bc43acf2e2173e3435056edd6843fc1646811e"} Oct 30 20:01:09.334843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.334819 4959 scope.go:122] "RemoveContainer" containerID="8583aa8a91bf97bb8b311b2559df04419a61ae4c9ac09d11785eea05ef6a06b9" Oct 30 20:01:09.375114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.375091 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 20:01:09.387864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.387841 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d96913-548c57d588-cbbwm"] Oct 30 20:01:09.423994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.423971 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 20:01:09.429724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.429702 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw"] Oct 30 20:01:09.538887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.538850 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:01:09.551878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:09.551856 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:01:10.001510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:10.001471 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" path="/var/lib/kubelet/pods/2cd1615f-32ce-49d5-ae8e-77ceeb1e5984/volumes" Oct 30 20:01:10.001996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:10.001979 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473aba10-bccb-47b9-b403-7fb253d16386" path="/var/lib/kubelet/pods/473aba10-bccb-47b9-b403-7fb253d16386/volumes" Oct 30 20:01:10.298798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:10.298725 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.467357 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:10.467329 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:01:10.643259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:10.643218 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2224e496-d58664b78-6hbxw" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.111:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:11.797063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:11.797030 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.547595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:12.547562 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.629965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:13.608247 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.794717 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.797223 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerName="maas-api" Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.797251 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerName="maas-api" Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.797390 4959 cpu_manager.go:414] "containerMap: removing container" podUID="473aba10-bccb-47b9-b403-7fb253d16386" containerName="payload-processing" Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.797401 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473aba10-bccb-47b9-b403-7fb253d16386" containerName="payload-processing" Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.798077 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="473aba10-bccb-47b9-b403-7fb253d16386" containerName="payload-processing" Oct 30 20:01:15.809727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.798137 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd1615f-32ce-49d5-ae8e-77ceeb1e5984" containerName="maas-api" Oct 30 20:01:15.929522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.902834 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:15.955519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:15.941812 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.041519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.026801 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ef132f-serving-cert\"" Oct 30 20:01:16.152929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.109413 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8c57\" (UniqueName: \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.392522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.308143 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.499576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.479449 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.524523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.522051 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.524523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.522237 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.636806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.636249 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.733905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.733859 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 20:01:16.759623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.748227 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded" Oct 30 20:01:16.865548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.865513 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.994517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.878599 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:16.994517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.880890 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.994517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.994216 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8c57\" (UniqueName: \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.994517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.994339 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.994517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.994390 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:16.994970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.948742 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:17.020405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:16.977802 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:17.020405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.017805 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:17.056540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.051590 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8c57\" (UniqueName: \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:17.058084 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:01:17.058055 4959 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s" Oct 30 20:01:17.058375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.058359 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:01:17.131589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.118087 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\") pod \"maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:17.131846 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:01:17.119852 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:01:17.132006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.129963 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.167944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.167918 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:17.650517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.644424 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.794751 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.792815 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.812778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.812753 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:01:17.862773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.862741 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:01:17.868872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.868851 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:17.886704 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.886667 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2224e496-dockercfg-dw9qp\"" Oct 30 20:01:17.887370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.887349 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:01:17.894168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.894099 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2224e496\"" Oct 30 20:01:17.954589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.926844 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqwb\" (UniqueName: \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:17.954589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:17.927090 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.120519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.115230 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.120519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.115590 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqwb\" (UniqueName: \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.129873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.121930 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.308679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.298539 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqwb\" (UniqueName: \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\") pod \"payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.376240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.376200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" event={"ID":"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac","Type":"ContainerStarted","Data":"d95f23e9633e2055a302bc29ae4d2adc2581f578f3f7fc0845c34e24d468f7de"} Oct 30 20:01:18.376401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.376299 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:01:18.417815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.417776 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:01:18.437017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.421158 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.438078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.428540 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spfnd\" (UniqueName: \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.438694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.438641 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.485621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.485595 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:18.540534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.540486 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spfnd\" (UniqueName: \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.540718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.540691 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.541546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.541528 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.565309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.565282 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spfnd\" (UniqueName: \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\") pod \"payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:18.787056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.787016 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.802103 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:01:18.802072 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d317f6_926d_43dd_a351_fa26a036e1fc.slice/crio-396fcdeb5d602531b18287aca4cfd8aa284f43a276b669803915df97240fc235 WatchSource:0}: Error finding container 396fcdeb5d602531b18287aca4cfd8aa284f43a276b669803915df97240fc235: Status 404 returned error can't find the container with id 396fcdeb5d602531b18287aca4cfd8aa284f43a276b669803915df97240fc235 Oct 30 20:01:18.804104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.804086 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:01:18.874885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:18.863813 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:19.296890 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:01:19.296855 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2ce641_95a5_4ecb_b66b_dadb43f5aace.slice/crio-05899e0a3ef5aa680bccc57630dff7ad178387be590b697afb29c91b02c24ed3 WatchSource:0}: Error finding container 05899e0a3ef5aa680bccc57630dff7ad178387be590b697afb29c91b02c24ed3: Status 404 returned error can't find the container with id 05899e0a3ef5aa680bccc57630dff7ad178387be590b697afb29c91b02c24ed3 Oct 30 20:01:19.297762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.297732 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:01:19.362647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.362613 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" event={"ID":"37d317f6-926d-43dd-a351-fa26a036e1fc","Type":"ContainerStarted","Data":"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985"} Oct 30 20:01:19.362819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.362680 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" event={"ID":"37d317f6-926d-43dd-a351-fa26a036e1fc","Type":"ContainerStarted","Data":"396fcdeb5d602531b18287aca4cfd8aa284f43a276b669803915df97240fc235"} Oct 30 20:01:19.362819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.362708 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:19.366556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.366528 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" event={"ID":"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac","Type":"ContainerStarted","Data":"a721002f79563d4540ab7d1ec15e603f1037fdce7178c268f3211ea30249b650"} Oct 30 20:01:19.366697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.366677 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:19.368093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.368069 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" event={"ID":"3f2ce641-95a5-4ecb-b66b-dadb43f5aace","Type":"ContainerStarted","Data":"05899e0a3ef5aa680bccc57630dff7ad178387be590b697afb29c91b02c24ed3"} Oct 30 20:01:19.409141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.409082 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" podStartSLOduration=2.40906411 podStartE2EDuration="2.40906411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:19.404933624 +0000 UTC m=+1603.534167546" watchObservedRunningTime="2026-10-30 20:01:19.40906411 +0000 UTC m=+1603.538298033" Oct 30 20:01:19.450147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:19.450093 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" podStartSLOduration=3.881481425 podStartE2EDuration="4.450067202s" totalImagesPullingTime="568.585777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:19.447392129 +0000 UTC m=+1603.576626059" watchObservedRunningTime="2026-10-30 20:01:19.450067202 +0000 UTC m=+1603.579301123" Oct 30 20:01:20.379200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:20.379164 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" event={"ID":"3f2ce641-95a5-4ecb-b66b-dadb43f5aace","Type":"ContainerStarted","Data":"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce"} Oct 30 20:01:20.379690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:20.379374 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:20.421834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:20.421724 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" podStartSLOduration=3.421710894 podStartE2EDuration="3.421710894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:20.419706493 +0000 UTC m=+1604.548940405" watchObservedRunningTime="2026-10-30 20:01:20.421710894 +0000 UTC m=+1604.550944815" Oct 30 20:01:20.427325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:20.427297 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.907827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:21.907781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.481673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:23.481636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:24.975813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:24.975782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.384379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:25.384355 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:01:26.578248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:26.578217 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.086693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:28.086661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.684612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:29.684579 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.165129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:30.165091 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.383811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:30.383784 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:01:31.217392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:31.217362 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.393436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:31.393413 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:01:32.831264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:32.831236 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.405644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:34.405610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.922198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:35.922169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:37.716748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:37.716718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.050673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:39.050630 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.478478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:39.478441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.208853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:41.208819 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:43.011146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:43.011117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.441860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:44.441824 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 20:01:44.442303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:44.442026 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" containerID="cri-o://a6890ac1f1119d8ddc9134dbfb8671b2d967825b59730f434655c986702520c3" gracePeriod=30 Oct 30 20:01:44.772723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:44.772635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.016174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:45.016142 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.096895 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:01:46.096861 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:01:46.972019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:46.971977 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.572932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:48.572901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.727625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.727594 4959 generic.go:356] "Generic (PLEG): container finished" podID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerID="a6890ac1f1119d8ddc9134dbfb8671b2d967825b59730f434655c986702520c3" exitCode=0 Oct 30 20:01:49.727625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.727654 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" event={"ID":"06298ea0-90dc-4a96-a90a-f9dde28fd618","Type":"ContainerDied","Data":"a6890ac1f1119d8ddc9134dbfb8671b2d967825b59730f434655c986702520c3"} Oct 30 20:01:49.727625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.727763 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" event={"ID":"06298ea0-90dc-4a96-a90a-f9dde28fd618","Type":"ContainerDied","Data":"dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab"} Oct 30 20:01:49.728280 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.727791 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd2be1437d78533915704b4d1ee859782bb265fc909d05aeecaf0102dabfacab" Oct 30 20:01:49.735183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.735161 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 20:01:49.839122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.839086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.856919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.856894 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.856938 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.856989 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857051 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857105 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857330 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:49.857433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857376 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857419 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:49.857560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857523 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857553 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857634 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\") pod \"06298ea0-90dc-4a96-a90a-f9dde28fd618\" (UID: \"06298ea0-90dc-4a96-a90a-f9dde28fd618\") " Oct 30 20:01:49.857837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857801 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:49.857922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.857874 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:49.858314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858249 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:49.858713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858690 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.858787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858720 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.858787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858735 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06298ea0-90dc-4a96-a90a-f9dde28fd618-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.858787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858751 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.858787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.858763 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.860011 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.859987 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:49.860118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.860009 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:49.860118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.860044 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:49.860229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.860118 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t" pod "06298ea0-90dc-4a96-a90a-f9dde28fd618" (UID: "06298ea0-90dc-4a96-a90a-f9dde28fd618"). InnerVolumeSpecName "kube-api-access-bjx8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:49.960875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.960791 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjx8t\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-kube-api-access-bjx8t\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.960875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.960821 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.960875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.960831 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:49.960875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:49.960840 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06298ea0-90dc-4a96-a90a-f9dde28fd618-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:01:50.000883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.000840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.184659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.184626 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.709180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.709145 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:01:50.709372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.709212 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:50.736296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.736265 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv" Oct 30 20:01:50.802036 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.802001 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 20:01:50.815590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:50.815559 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-t9zcv"] Oct 30 20:01:51.374096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:51.374056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.520521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:51.514420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.748987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:51.748960 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.099570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:52.095444 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" path="/var/lib/kubelet/pods/06298ea0-90dc-4a96-a90a-f9dde28fd618/volumes" Oct 30 20:01:53.407724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:53.407684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.142640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:54.142605 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.260879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:54.260844 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.043520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:55.035180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.720068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:56.720025 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.849554 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:56.837216 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.201102 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:57.201068 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.716077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:57.716038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.025426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:58.025309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.899463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:01:58.899403 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.656440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:00.656363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.724591 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:01.724552 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.890050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:01.890014 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:02.886420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:02.886379 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.915518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:03.900964 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.075122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:04.075086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.031484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.031458 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:02:05.045229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.033175 4959 cpu_manager.go:414] "containerMap: removing container" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" Oct 30 20:02:05.045229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.041255 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" Oct 30 20:02:05.045229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.041591 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="06298ea0-90dc-4a96-a90a-f9dde28fd618" containerName="istio-proxy" Oct 30 20:02:05.056325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.056300 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:02:05.056877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.056408 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.059518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.059005 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:02:05.064945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.064799 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.071972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.071947 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:02:05.074260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.074240 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-936f70\"" Oct 30 20:02:05.088432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.088417 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-936f70-dockercfg-8g9gc\"" Oct 30 20:02:05.224292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.224247 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7lf\" (UniqueName: \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.224484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.224375 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.224619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.224536 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.224685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.224645 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w54\" (UniqueName: \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.326487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.326394 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7lf\" (UniqueName: \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.326487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.326448 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.326761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.326555 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.326761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.326653 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w54\" (UniqueName: \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.327324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.327297 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.327547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.327488 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.348206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.348174 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7lf\" (UniqueName: \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\") pod \"payload-processing-e2e-inf-b-936f70-78f8989876-wmljt\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.348337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.348284 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w54\" (UniqueName: \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\") pod \"payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.384918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.384888 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:05.392374 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.392337 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:05.712317 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:02:05.712286 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad91e702_e7fc_4865_9ae6_1e61b631b593.slice/crio-c1692488da5d73d00dfaa82d84ee4bef70fa2ef6ac68fb4252173420daa10d4e WatchSource:0}: Error finding container c1692488da5d73d00dfaa82d84ee4bef70fa2ef6ac68fb4252173420daa10d4e: Status 404 returned error can't find the container with id c1692488da5d73d00dfaa82d84ee4bef70fa2ef6ac68fb4252173420daa10d4e Oct 30 20:02:05.715335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.715316 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:02:05.734347 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:02:05.734319 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99fd189_0734_45dc_ac8a_f7f526288126.slice/crio-c138235bdbf9bf2e301979914e10742296e27affea7fac0d2faedf44ac76349e WatchSource:0}: Error finding container c138235bdbf9bf2e301979914e10742296e27affea7fac0d2faedf44ac76349e: Status 404 returned error can't find the container with id c138235bdbf9bf2e301979914e10742296e27affea7fac0d2faedf44ac76349e Oct 30 20:02:05.734965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.734924 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:02:05.823231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:05.823199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.066002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.065903 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" event={"ID":"e99fd189-0734-45dc-ac8a-f7f526288126","Type":"ContainerStarted","Data":"bbaae0d3120ad307c9b0ad5cf12ac3b121c6ffa08bceae285ea6777bba01ca7b"} Oct 30 20:02:06.066002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.065967 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" event={"ID":"e99fd189-0734-45dc-ac8a-f7f526288126","Type":"ContainerStarted","Data":"c138235bdbf9bf2e301979914e10742296e27affea7fac0d2faedf44ac76349e"} Oct 30 20:02:06.066002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.066001 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:06.067878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.067848 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" event={"ID":"ad91e702-e7fc-4865-9ae6-1e61b631b593","Type":"ContainerStarted","Data":"ab98eadada824542133f092e3dd26cb06204ec4d18b70a599b6c4714cc98d1f3"} Oct 30 20:02:06.067989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.067916 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:06.067989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.067945 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" event={"ID":"ad91e702-e7fc-4865-9ae6-1e61b631b593","Type":"ContainerStarted","Data":"c1692488da5d73d00dfaa82d84ee4bef70fa2ef6ac68fb4252173420daa10d4e"} Oct 30 20:02:06.115728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.115680 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" podStartSLOduration=2.115666424 podStartE2EDuration="2.115666424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:06.113714193 +0000 UTC m=+1650.242948118" watchObservedRunningTime="2026-10-30 20:02:06.115666424 +0000 UTC m=+1650.244900390" Oct 30 20:02:06.157366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:06.157316 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" podStartSLOduration=2.157298545 podStartE2EDuration="2.157298545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:06.155597061 +0000 UTC m=+1650.284830974" watchObservedRunningTime="2026-10-30 20:02:06.157298545 +0000 UTC m=+1650.286532512" Oct 30 20:02:07.687920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:07.687870 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.094894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:08.094816 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.295729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:08.290115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.241952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:09.241917 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.066522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:11.036422 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.526427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:12.526388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.553855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:13.553826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.810289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:13.810187 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.937698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:13.931911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.075683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:14.075596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.187613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:14.187583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.382035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:14.381962 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.702545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.702515 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.809338 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.809305 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:02:15.818332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.818300 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:15.824378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.824347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.832373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.832146 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:02:15.839461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.839436 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7b5635\"" Oct 30 20:02:15.920351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.920308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ng8r\" (UniqueName: \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:15.920728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:15.920702 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.060311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.052825 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ng8r\" (UniqueName: \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.060311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.053083 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.060311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.054072 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.125232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.125195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ng8r\" (UniqueName: \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\") pod \"payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.298060 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:16.297899 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:02:16.440205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.440176 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:16.748563 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:02:16.748536 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf556501d_9a01_405a_8fde_af056b3af550.slice/crio-a244f5dda40a583fd534c83dbd5d353adc77479a27c003b3ba905ba2ce185b7d WatchSource:0}: Error finding container a244f5dda40a583fd534c83dbd5d353adc77479a27c003b3ba905ba2ce185b7d: Status 404 returned error can't find the container with id a244f5dda40a583fd534c83dbd5d353adc77479a27c003b3ba905ba2ce185b7d Oct 30 20:02:16.749906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:16.749883 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:02:17.079372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.079298 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:02:17.079372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.079360 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:02:17.258945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.258910 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" event={"ID":"f556501d-9a01-405a-8fde-af056b3af550","Type":"ContainerStarted","Data":"cee95ccf7aab3675b2719a47f0f2553000c4f6fb99b385a41d0b3312c2a8b054"} Oct 30 20:02:17.259113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.258973 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" event={"ID":"f556501d-9a01-405a-8fde-af056b3af550","Type":"ContainerStarted","Data":"a244f5dda40a583fd534c83dbd5d353adc77479a27c003b3ba905ba2ce185b7d"} Oct 30 20:02:17.259113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.259005 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:17.307318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.307255 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" podStartSLOduration=2.307234971 podStartE2EDuration="2.307234971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:17.303069201 +0000 UTC m=+1661.432303136" watchObservedRunningTime="2026-10-30 20:02:17.307234971 +0000 UTC m=+1661.436468893" Oct 30 20:02:17.499205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:17.499168 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.221019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:19.220990 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.750091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:20.750058 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 20:02:20.750607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:20.750306 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" podUID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" containerName="maas-api" containerID="cri-o://a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff" gracePeriod=30 Oct 30 20:02:20.811305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:20.811247 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.089525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.088630 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 20:02:21.193520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.184738 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") pod \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " Oct 30 20:02:21.193520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.184820 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\" (UniqueName: \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\") pod \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " Oct 30 20:02:21.193520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.184886 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\") pod \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " Oct 30 20:02:21.193520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.184936 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\") pod \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\" (UID: \"bde97bd8-1596-4f6d-8091-6fcf87f44da1\") " Oct 30 20:02:21.193520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.186414 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca" pod "bde97bd8-1596-4f6d-8091-6fcf87f44da1" (UID: "bde97bd8-1596-4f6d-8091-6fcf87f44da1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:21.202515 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.199710 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls" pod "bde97bd8-1596-4f6d-8091-6fcf87f44da1" (UID: "bde97bd8-1596-4f6d-8091-6fcf87f44da1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:21.240525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.231284 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n" pod "bde97bd8-1596-4f6d-8091-6fcf87f44da1" (UID: "bde97bd8-1596-4f6d-8091-6fcf87f44da1"). InnerVolumeSpecName "kube-api-access-kws5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:21.265128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.248987 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs" pod "bde97bd8-1596-4f6d-8091-6fcf87f44da1" (UID: "bde97bd8-1596-4f6d-8091-6fcf87f44da1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:21.468566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.457610 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.468566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.457661 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kws5n\" (UniqueName: \"kubernetes.io/projected/bde97bd8-1596-4f6d-8091-6fcf87f44da1-kube-api-access-kws5n\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.468566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.457687 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde97bd8-1596-4f6d-8091-6fcf87f44da1-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.468566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.457702 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde97bd8-1596-4f6d-8091-6fcf87f44da1-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:21.690518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.672952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.784897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.784862 4959 generic.go:356] "Generic (PLEG): container finished" podID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" containerID="a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff" exitCode=0 Oct 30 20:02:21.785237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.785193 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" event={"ID":"bde97bd8-1596-4f6d-8091-6fcf87f44da1","Type":"ContainerDied","Data":"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff"} Oct 30 20:02:21.785279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.785246 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" event={"ID":"bde97bd8-1596-4f6d-8091-6fcf87f44da1","Type":"ContainerDied","Data":"7eea5af8956e899efa11911a62c2c543a704304ad2343d1fde51ec630cddee6d"} Oct 30 20:02:21.785315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.785280 4959 scope.go:122] "RemoveContainer" containerID="a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff" Oct 30 20:02:21.785964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.785448 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7" Oct 30 20:02:21.816925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.816904 4959 scope.go:122] "RemoveContainer" containerID="a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff" Oct 30 20:02:21.817299 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:21.817272 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff\": container with ID starting with a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff not found: ID does not exist" containerID="a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff" Oct 30 20:02:21.817379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.817303 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff"} err="failed to get container status \"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff\": rpc error: code = NotFound desc = could not find container \"a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff\": container with ID starting with a0acf69400d88c196544fba300c1e3af55219ffb809eca683e76e0fff65097ff not found: ID does not exist" Oct 30 20:02:21.825486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.825457 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.880344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.880296 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 20:02:21.893222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:21.887234 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-540a43-6d6c56d8cc-wqhx7"] Oct 30 20:02:22.032573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:22.029797 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.081561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:22.076220 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" path="/var/lib/kubelet/pods/bde97bd8-1596-4f6d-8091-6fcf87f44da1/volumes" Oct 30 20:02:22.270118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:22.270088 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.686522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:22.678260 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.800825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:22.800790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.078339 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:23.078268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.192358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:23.192323 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:24.162522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:24.156974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:25.996627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:25.996594 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.467517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:27.467455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.664780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:27.663852 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.857111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:27.857026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.122409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:28.122316 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.270231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:28.270205 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:02:29.991950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:29.991899 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:30.100537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.100480 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:30.166813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.166780 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:02:30.167689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.167670 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" containerName="maas-api" Oct 30 20:02:30.167689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.167687 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" containerName="maas-api" Oct 30 20:02:30.167889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.167832 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde97bd8-1596-4f6d-8091-6fcf87f44da1" containerName="maas-api" Oct 30 20:02:30.171913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.171890 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.189588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.189564 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:02:30.196969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.196945 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c17154\"" Oct 30 20:02:30.220422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.220384 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpls\" (UniqueName: \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.225827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.225785 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.289569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.277887 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c17154-dockercfg-qhbjd\"" Oct 30 20:02:30.398518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.396972 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpls\" (UniqueName: \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.398518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.397061 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.447523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.439328 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.480221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.480184 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpls\" (UniqueName: \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\") pod \"payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.485661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.485633 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:30.501530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.500511 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:30.642372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.593736 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:02:30.668181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.667939 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.676055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.676021 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:02:30.730543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.725696 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvws\" (UniqueName: \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.730543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.726610 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.828506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.828457 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvws\" (UniqueName: \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.828705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.828548 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.829304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.829281 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.850364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.850334 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvws\" (UniqueName: \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\") pod \"payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:30.993269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:30.993237 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:02:31.031130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:31.031092 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:31.342142 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:02:31.342112 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740a4953_6deb_48e4_90cd_b5d09c1b9706.slice/crio-ba1486b74c97861e0a6b25bd5a0db3d51bba4dd002559ff128262fc1d497397b WatchSource:0}: Error finding container ba1486b74c97861e0a6b25bd5a0db3d51bba4dd002559ff128262fc1d497397b: Status 404 returned error can't find the container with id ba1486b74c97861e0a6b25bd5a0db3d51bba4dd002559ff128262fc1d497397b Oct 30 20:02:31.343244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:31.343219 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:02:31.360259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:31.360225 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.531525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:31.530511 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.749519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:31.748569 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.101572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.100223 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:32.101572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.100281 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" event={"ID":"d7c3d7f6-14af-433c-9db2-553060cfa7cc","Type":"ContainerStarted","Data":"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5"} Oct 30 20:02:32.101572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.100314 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" event={"ID":"d7c3d7f6-14af-433c-9db2-553060cfa7cc","Type":"ContainerStarted","Data":"42c2242d347bb829d0b8fae7111ecaca40f40a18ab25a55b37e7c602a0c34ad9"} Oct 30 20:02:32.107071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.107036 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" event={"ID":"740a4953-6deb-48e4-90cd-b5d09c1b9706","Type":"ContainerStarted","Data":"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b"} Oct 30 20:02:32.107313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.107291 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" event={"ID":"740a4953-6deb-48e4-90cd-b5d09c1b9706","Type":"ContainerStarted","Data":"ba1486b74c97861e0a6b25bd5a0db3d51bba4dd002559ff128262fc1d497397b"} Oct 30 20:02:32.108703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.108683 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:32.200633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.200571 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" podStartSLOduration=2.200538167 podStartE2EDuration="2.200538167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:32.126456915 +0000 UTC m=+1676.255690837" watchObservedRunningTime="2026-10-30 20:02:32.200538167 +0000 UTC m=+1676.329772088" Oct 30 20:02:32.210113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.210051 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" podStartSLOduration=2.210031652 podStartE2EDuration="2.210031652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:32.202035262 +0000 UTC m=+1676.331269184" watchObservedRunningTime="2026-10-30 20:02:32.210031652 +0000 UTC m=+1676.339265573" Oct 30 20:02:32.387109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.387076 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.554409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:32.554375 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.148378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:33.148332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.332219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:33.332182 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.789789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:34.789716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:35.261557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.261527 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7062-a54173"] Oct 30 20:02:35.266519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.266472 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:35.282335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.282310 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\"" Oct 30 20:02:35.283347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.283326 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7062-a54173"] Oct 30 20:02:35.364470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.364440 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn97r\" (UniqueName: \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\") pod \"test-curl-7062-a54173\" (UID: \"68744e13-fc8d-4aac-9d7d-834849bbd85b\") " pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:35.466183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.466151 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn97r\" (UniqueName: \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\") pod \"test-curl-7062-a54173\" (UID: \"68744e13-fc8d-4aac-9d7d-834849bbd85b\") " pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:35.487718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.487683 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn97r\" (UniqueName: \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\") pod \"test-curl-7062-a54173\" (UID: \"68744e13-fc8d-4aac-9d7d-834849bbd85b\") " pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:35.581431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.581356 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:35.913848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:35.913815 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7062-a54173"] Oct 30 20:02:36.143410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.143377 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7062-a54173" event={"ID":"68744e13-fc8d-4aac-9d7d-834849bbd85b","Type":"ContainerStarted","Data":"8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86"} Oct 30 20:02:36.143628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.143431 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7062-a54173" event={"ID":"68744e13-fc8d-4aac-9d7d-834849bbd85b","Type":"ContainerStarted","Data":"09d51898e4a590c754b0490bb6bb6523f559851e26fdc6f74fcdd3ccb2a51cd3"} Oct 30 20:02:36.187586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.187348 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7062-a54173" podStartSLOduration=1.187329999 podStartE2EDuration="1.187329999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:36.185148222 +0000 UTC m=+1680.314382143" watchObservedRunningTime="2026-10-30 20:02:36.187329999 +0000 UTC m=+1680.316563920" Oct 30 20:02:36.638161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.638128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.693563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.693526 4959 scope.go:122] "RemoveContainer" containerID="a6890ac1f1119d8ddc9134dbfb8671b2d967825b59730f434655c986702520c3" Oct 30 20:02:36.746805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:36.746779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.386192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:37.386162 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7062-a54173"] Oct 30 20:02:37.394559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:37.394528 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7062-a54173"] Oct 30 20:02:38.231386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:38.231348 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7062-a54173" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" containerName="test-curl-7062-a54173" containerID="cri-o://8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86" gracePeriod=2 Oct 30 20:02:38.502797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:38.502716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.619340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:38.619301 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.734842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:38.734808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.467064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:40.467036 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:40.480669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:40.480631 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\" (UniqueName: \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\") pod \"68744e13-fc8d-4aac-9d7d-834849bbd85b\" (UID: \"68744e13-fc8d-4aac-9d7d-834849bbd85b\") " Oct 30 20:02:40.482838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:40.482812 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r" pod "68744e13-fc8d-4aac-9d7d-834849bbd85b" (UID: "68744e13-fc8d-4aac-9d7d-834849bbd85b"). InnerVolumeSpecName "kube-api-access-nn97r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:40.582890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:40.582866 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn97r\" (UniqueName: \"kubernetes.io/projected/68744e13-fc8d-4aac-9d7d-834849bbd85b-kube-api-access-nn97r\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:40.686870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:40.686839 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.259461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:41.259433 4959 generic.go:356] "Generic (PLEG): container finished" podID="68744e13-fc8d-4aac-9d7d-834849bbd85b" containerID="8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86" exitCode=137 Oct 30 20:02:41.259666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:41.259487 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:41.259666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:41.259539 4959 scope.go:122] "RemoveContainer" containerID="8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86" Oct 30 20:02:41.270526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:41.270483 4959 scope.go:122] "RemoveContainer" containerID="8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86" Oct 30 20:02:41.270797 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:41.270779 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86\": container with ID starting with 8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86 not found: ID does not exist" containerID="8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86" Oct 30 20:02:41.270880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:41.270803 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86"} err="failed to get container status \"8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86\": rpc error: code = NotFound desc = could not find container \"8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86\": container with ID starting with 8c5c4cd48ca9521d0137b77f62059886ea6287ee99f8381c3d719c633c772f86 not found: ID does not exist" Oct 30 20:02:41.284777 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:41.284753 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7062-a54173\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" pod="opendatahub/test-curl-7062-a54173" Oct 30 20:02:42.000824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:42.000789 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" path="/var/lib/kubelet/pods/68744e13-fc8d-4aac-9d7d-834849bbd85b/volumes" Oct 30 20:02:42.476564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:42.476486 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.121944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:43.121916 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:02:44.066091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.066057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.154670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.153946 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:02:44.295390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.295359 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 20:02:44.295612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.295586 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" containerID="cri-o://cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762" gracePeriod=30 Oct 30 20:02:44.800922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.800891 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 20:02:44.801108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:44.801086 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" containerID="cri-o://453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510" gracePeriod=30 Oct 30 20:02:45.580628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:45.580592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.066481 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:46.066447 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:02:47.232437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.232382 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.449508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.449451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.603232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.603143 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.781922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.781888 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:02:47.816635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.816608 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:02:47.925570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:47.925535 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:48.072515 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:48.072464 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.139433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.139398 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:02:49.139868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.139464 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:02:49.251574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.249716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.602659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.602636 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 20:02:49.608028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608001 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608039 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608091 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608149 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608280 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608297 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.608333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608309 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.608405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608364 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608387 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608507 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.608596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608546 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608590 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.608636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608621 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\") pod \"01d7759f-78c0-4427-8451-fe3022de4942\" (UID: \"01d7759f-78c0-4427-8451-fe3022de4942\") " Oct 30 20:02:49.608914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.608887 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:49.609718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.609692 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.609722 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01d7759f-78c0-4427-8451-fe3022de4942-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.609742 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.609757 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.609846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.609772 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.610760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.610738 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:49.610855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.610762 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.610855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.610793 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "kube-api-access-h9xhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:49.610946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.610930 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy" pod "01d7759f-78c0-4427-8451-fe3022de4942" (UID: "01d7759f-78c0-4427-8451-fe3022de4942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:49.711154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.711073 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01d7759f-78c0-4427-8451-fe3022de4942-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.711154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.711106 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xhd\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-kube-api-access-h9xhd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.711154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.711118 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01d7759f-78c0-4427-8451-fe3022de4942-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:49.711154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:49.711130 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01d7759f-78c0-4427-8451-fe3022de4942-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.113377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.113349 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 20:02:50.243075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243035 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243144 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243198 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243320 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243411 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243565 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243679 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243718 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243776 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.243918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243801 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\") pod \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\" (UID: \"15e682b0-f4bd-403b-b358-73e30d2d6b3e\") " Oct 30 20:02:50.243918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.243803 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.244179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244015 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.244179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244107 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:50.244681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244654 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.244814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244687 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.244814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244702 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.244814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.244716 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.248897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.247451 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.248897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.248205 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.248897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.248427 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "kube-api-access-btcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:50.248897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.248588 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:50.249414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.249390 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo" pod "15e682b0-f4bd-403b-b358-73e30d2d6b3e" (UID: "15e682b0-f4bd-403b-b358-73e30d2d6b3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:50.346239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.346117 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.346239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.346150 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.346239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.346166 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e682b0-f4bd-403b-b358-73e30d2d6b3e-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.346239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.346180 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcgd\" (UniqueName: \"kubernetes.io/projected/15e682b0-f4bd-403b-b358-73e30d2d6b3e-kube-api-access-btcgd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.346239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.346195 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e682b0-f4bd-403b-b358-73e30d2d6b3e-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:50.385926 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.385896 4959 generic.go:356] "Generic (PLEG): container finished" podID="01d7759f-78c0-4427-8451-fe3022de4942" containerID="cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762" exitCode=0 Oct 30 20:02:50.386072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.385964 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" Oct 30 20:02:50.386072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.385975 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerDied","Data":"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762"} Oct 30 20:02:50.386072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.386026 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf" event={"ID":"01d7759f-78c0-4427-8451-fe3022de4942","Type":"ContainerDied","Data":"9ba410b13c963b502a3b7980f09bd414527642e7a7ceb28f1cbed45a9db3123f"} Oct 30 20:02:50.386072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.386051 4959 scope.go:122] "RemoveContainer" containerID="cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762" Oct 30 20:02:50.387451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.387428 4959 generic.go:356] "Generic (PLEG): container finished" podID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerID="453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510" exitCode=0 Oct 30 20:02:50.387592 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.387508 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" Oct 30 20:02:50.387712 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.387507 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" event={"ID":"15e682b0-f4bd-403b-b358-73e30d2d6b3e","Type":"ContainerDied","Data":"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510"} Oct 30 20:02:50.387782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.387760 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj" event={"ID":"15e682b0-f4bd-403b-b358-73e30d2d6b3e","Type":"ContainerDied","Data":"ff0cab8426a44c8fd3c7ee40f3221324a489341a8aa173e6ff28cebc1dbfd285"} Oct 30 20:02:50.397540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.397482 4959 scope.go:122] "RemoveContainer" containerID="db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab" Oct 30 20:02:50.410634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.410614 4959 scope.go:122] "RemoveContainer" containerID="cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762" Oct 30 20:02:50.410912 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:50.410889 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762\": container with ID starting with cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762 not found: ID does not exist" containerID="cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762" Oct 30 20:02:50.411005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.410919 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762"} err="failed to get container status \"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762\": rpc error: code = NotFound desc = could not find container \"cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762\": container with ID starting with cbdf62cb0c3224c4e5f6b74853c01e5baab91724373f0a4697b95f0a62249762 not found: ID does not exist" Oct 30 20:02:50.411005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.410936 4959 scope.go:122] "RemoveContainer" containerID="db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab" Oct 30 20:02:50.411164 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:50.411149 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab\": container with ID starting with db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab not found: ID does not exist" containerID="db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab" Oct 30 20:02:50.411202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.411168 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab"} err="failed to get container status \"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab\": rpc error: code = NotFound desc = could not find container \"db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab\": container with ID starting with db2f40ac47a311f398f648b52e5823f1cc6f7151905a5e5967f146826742a0ab not found: ID does not exist" Oct 30 20:02:50.411202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.411180 4959 scope.go:122] "RemoveContainer" containerID="453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510" Oct 30 20:02:50.434771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.431670 4959 scope.go:122] "RemoveContainer" containerID="453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510" Oct 30 20:02:50.434771 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:50.432300 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510\": container with ID starting with 453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510 not found: ID does not exist" containerID="453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510" Oct 30 20:02:50.434771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.432334 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510"} err="failed to get container status \"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510\": rpc error: code = NotFound desc = could not find container \"453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510\": container with ID starting with 453513827f066da0720f68604c282620691fbcc8cd9253fb90d65f0543b4c510 not found: ID does not exist" Oct 30 20:02:50.455235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.455211 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 20:02:50.459382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.459360 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-jsmmf"] Oct 30 20:02:50.494861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.494823 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 20:02:50.498792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.498761 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-qjfsj"] Oct 30 20:02:50.726880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:50.726849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.001268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:52.001232 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d7759f-78c0-4427-8451-fe3022de4942" path="/var/lib/kubelet/pods/01d7759f-78c0-4427-8451-fe3022de4942/volumes" Oct 30 20:02:52.001980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:52.001963 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" path="/var/lib/kubelet/pods/15e682b0-f4bd-403b-b358-73e30d2d6b3e/volumes" Oct 30 20:02:52.293857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:52.293756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.783035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:52.782994 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" podUID="37d317f6-926d-43dd-a351-fa26a036e1fc" containerName="payload-processing" containerID="cri-o://fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985" gracePeriod=25 Oct 30 20:02:52.817098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:52.817047 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" podUID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" containerName="payload-pre-processing" containerID="cri-o://e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce" gracePeriod=25 Oct 30 20:02:53.120842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.120817 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:02:53.178135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.178110 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:02:53.207506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.207452 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\") pod \"37d317f6-926d-43dd-a351-fa26a036e1fc\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " Oct 30 20:02:53.207723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.207579 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\" (UniqueName: \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\") pod \"37d317f6-926d-43dd-a351-fa26a036e1fc\" (UID: \"37d317f6-926d-43dd-a351-fa26a036e1fc\") " Oct 30 20:02:53.207936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.207908 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume" pod "37d317f6-926d-43dd-a351-fa26a036e1fc" (UID: "37d317f6-926d-43dd-a351-fa26a036e1fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:53.208363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.208343 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37d317f6-926d-43dd-a351-fa26a036e1fc-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:53.209920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.209898 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb" pod "37d317f6-926d-43dd-a351-fa26a036e1fc" (UID: "37d317f6-926d-43dd-a351-fa26a036e1fc"). InnerVolumeSpecName "kube-api-access-fpqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:53.309803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.309770 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\") pod \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " Oct 30 20:02:53.310110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.310095 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\" (UniqueName: \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\") pod \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\" (UID: \"3f2ce641-95a5-4ecb-b66b-dadb43f5aace\") " Oct 30 20:02:53.310110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.310097 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume" pod "3f2ce641-95a5-4ecb-b66b-dadb43f5aace" (UID: "3f2ce641-95a5-4ecb-b66b-dadb43f5aace"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:53.310570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.310552 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:53.310656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.310574 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqwb\" (UniqueName: \"kubernetes.io/projected/37d317f6-926d-43dd-a351-fa26a036e1fc-kube-api-access-fpqwb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:53.312695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.312669 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd" pod "3f2ce641-95a5-4ecb-b66b-dadb43f5aace" (UID: "3f2ce641-95a5-4ecb-b66b-dadb43f5aace"). InnerVolumeSpecName "kube-api-access-spfnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:53.412145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.412116 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spfnd\" (UniqueName: \"kubernetes.io/projected/3f2ce641-95a5-4ecb-b66b-dadb43f5aace-kube-api-access-spfnd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:02:53.450224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.450192 4959 generic.go:356] "Generic (PLEG): container finished" podID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" containerID="e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce" exitCode=0 Oct 30 20:02:53.450386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.450248 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" Oct 30 20:02:53.450386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.450274 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" event={"ID":"3f2ce641-95a5-4ecb-b66b-dadb43f5aace","Type":"ContainerDied","Data":"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce"} Oct 30 20:02:53.450386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.450345 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr" event={"ID":"3f2ce641-95a5-4ecb-b66b-dadb43f5aace","Type":"ContainerDied","Data":"05899e0a3ef5aa680bccc57630dff7ad178387be590b697afb29c91b02c24ed3"} Oct 30 20:02:53.450386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.450376 4959 scope.go:122] "RemoveContainer" containerID="e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce" Oct 30 20:02:53.451983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.451959 4959 generic.go:356] "Generic (PLEG): container finished" podID="37d317f6-926d-43dd-a351-fa26a036e1fc" containerID="fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985" exitCode=0 Oct 30 20:02:53.452082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.452010 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" Oct 30 20:02:53.452156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.452092 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" event={"ID":"37d317f6-926d-43dd-a351-fa26a036e1fc","Type":"ContainerDied","Data":"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985"} Oct 30 20:02:53.452156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.452131 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288" event={"ID":"37d317f6-926d-43dd-a351-fa26a036e1fc","Type":"ContainerDied","Data":"396fcdeb5d602531b18287aca4cfd8aa284f43a276b669803915df97240fc235"} Oct 30 20:02:53.461766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.461752 4959 scope.go:122] "RemoveContainer" containerID="e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce" Oct 30 20:02:53.462049 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:53.462024 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce\": container with ID starting with e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce not found: ID does not exist" containerID="e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce" Oct 30 20:02:53.462162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.462054 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce"} err="failed to get container status \"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce\": rpc error: code = NotFound desc = could not find container \"e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce\": container with ID starting with e5bc4b30dd5f1e8568300d17e9a4980e06977ce2d1b65e3223740773358528ce not found: ID does not exist" Oct 30 20:02:53.462162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.462078 4959 scope.go:122] "RemoveContainer" containerID="fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985" Oct 30 20:02:53.472658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.472640 4959 scope.go:122] "RemoveContainer" containerID="fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985" Oct 30 20:02:53.472952 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:02:53.472934 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985\": container with ID starting with fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985 not found: ID does not exist" containerID="fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985" Oct 30 20:02:53.473016 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.472963 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985"} err="failed to get container status \"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985\": rpc error: code = NotFound desc = could not find container \"fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985\": container with ID starting with fc97d6d9597d73ecf956314d5a8af10c3635857fd60ec72d9ef6ecc815c49985 not found: ID does not exist" Oct 30 20:02:53.508108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.508080 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:02:53.520975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.520950 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2224e496-fb4fcd48c-r8vnr"] Oct 30 20:02:53.548719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.548695 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:02:53.551996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.551975 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2224e496-fb5cd48cb-xn288"] Oct 30 20:02:53.788619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:53.788535 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.226518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:54.218674 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d317f6-926d-43dd-a351-fa26a036e1fc" path="/var/lib/kubelet/pods/37d317f6-926d-43dd-a351-fa26a036e1fc/volumes" Oct 30 20:02:54.226518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:54.219447 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" path="/var/lib/kubelet/pods/3f2ce641-95a5-4ecb-b66b-dadb43f5aace/volumes" Oct 30 20:02:55.238250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:55.238205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.810522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:55.808766 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.112882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:56.112779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.222885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:56.222848 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.592341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:56.592291 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.175516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.175430 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176322 4959 cpu_manager.go:414] "containerMap: removing container" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176339 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176350 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" containerName="payload-pre-processing" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176355 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" containerName="payload-pre-processing" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176381 4959 cpu_manager.go:414] "containerMap: removing container" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" containerName="test-curl-7062-a54173" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176386 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" containerName="test-curl-7062-a54173" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176399 4959 cpu_manager.go:414] "containerMap: removing container" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176404 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176411 4959 cpu_manager.go:414] "containerMap: removing container" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176417 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176427 4959 cpu_manager.go:414] "containerMap: removing container" podUID="37d317f6-926d-43dd-a351-fa26a036e1fc" containerName="payload-processing" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176431 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d317f6-926d-43dd-a351-fa26a036e1fc" containerName="payload-processing" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176601 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176611 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2ce641-95a5-4ecb-b66b-dadb43f5aace" containerName="payload-pre-processing" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176627 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="68744e13-fc8d-4aac-9d7d-834849bbd85b" containerName="test-curl-7062-a54173" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176640 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e682b0-f4bd-403b-b358-73e30d2d6b3e" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176645 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d7759f-78c0-4427-8451-fe3022de4942" containerName="istio-proxy" Oct 30 20:02:58.179547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.176651 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d317f6-926d-43dd-a351-fa26a036e1fc" containerName="payload-processing" Oct 30 20:02:58.180833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.180814 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.197619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.197599 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:02:58.277522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.277441 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7b5635-serving-cert\"" Oct 30 20:02:58.281811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.281780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.307203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.307169 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.307342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.307248 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46xz\" (UniqueName: \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.307541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.307522 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.307688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.307671 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.401613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.401583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.411206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.410995 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.411356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.411246 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.411356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.411304 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46xz\" (UniqueName: \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.411721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.411606 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.411863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.411841 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.414277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.414249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.415074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.415049 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.437183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.437117 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46xz\" (UniqueName: \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\") pod \"maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.531540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.530294 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:02:58.542038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:58.542009 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:59.073553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:59.053452 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:02:59.617044 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:02:59.617010 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" event={"ID":"6c62448d-e119-4af6-904e-0660b3bb00de","Type":"ContainerStarted","Data":"76a9d1868ee362578a43c49161c7c4c3d73ce8b9924a10d0a5ad736258309228"} Oct 30 20:03:00.207778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:00.207727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:00.626770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:00.626675 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" event={"ID":"6c62448d-e119-4af6-904e-0660b3bb00de","Type":"ContainerStarted","Data":"dcceb0fc9b37a572108541821ddaaca97510027e26aadcdff3f12ae94f758b9b"} Oct 30 20:03:00.627264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:00.626945 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:03:00.679046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:00.678990 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" podStartSLOduration=2.132042688 podStartE2EDuration="2.67897307s" totalImagesPullingTime="546.930382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:00.673142658 +0000 UTC m=+1704.802376579" watchObservedRunningTime="2026-10-30 20:03:00.67897307 +0000 UTC m=+1704.808207000" Oct 30 20:03:01.822812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:01.822777 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.398775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:03.398735 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.924214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:03.924179 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.058524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:04.055883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.308049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:04.307967 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:05.265606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:05.265578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.639476 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:06.639444 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:03:06.954116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:06.953983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:08.686166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:08.686134 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.373447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:10.373404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.489942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:10.489908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.603036 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:10.602992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.299786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:12.299730 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:13.910716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:13.910676 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.796067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:14.796006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.889099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:14.889071 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 20:03:14.889282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:14.889262 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" containerID="cri-o://1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911" gracePeriod=30 Oct 30 20:03:14.930989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:14.930955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.051046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:15.050955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.337009 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:15.336922 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.608472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:15.608308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.102737 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:03:16.102701 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:03:16.809586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:16.809556 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:18.434214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:18.434170 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:18.564516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:18.563282 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:18.756624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:18.756475 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.250466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.242458 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 20:03:20.277457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.277367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.383195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383151 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383231 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383303 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383344 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383400 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383563 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383617 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383720 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.383879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.383844 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\") pod \"ff97fb2a-8949-4354-a737-0b40e026b6b1\" (UID: \"ff97fb2a-8949-4354-a737-0b40e026b6b1\") " Oct 30 20:03:20.384917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.384886 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.385331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.385305 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.385469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.385321 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.385620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.385602 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:20.385763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.385745 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.391964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.391891 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "kube-api-access-spcx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.392091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.392026 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:20.393891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.392653 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:20.395343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.395305 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy" pod "ff97fb2a-8949-4354-a737-0b40e026b6b1" (UID: "ff97fb2a-8949-4354-a737-0b40e026b6b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:20.463477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.463396 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 20:03:20.486404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486375 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486402 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97fb2a-8949-4354-a737-0b40e026b6b1-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486412 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spcx7\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-kube-api-access-spcx7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486420 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486430 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486439 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486448 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486457 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.486644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.486466 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97fb2a-8949-4354-a737-0b40e026b6b1-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:20.555624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.555591 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.828757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.828679 4959 generic.go:356] "Generic (PLEG): container finished" podID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerID="1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911" exitCode=0 Oct 30 20:03:20.828757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.828712 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" event={"ID":"ff97fb2a-8949-4354-a737-0b40e026b6b1","Type":"ContainerDied","Data":"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911"} Oct 30 20:03:20.828757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.828730 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" Oct 30 20:03:20.828757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.828749 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7" event={"ID":"ff97fb2a-8949-4354-a737-0b40e026b6b1","Type":"ContainerDied","Data":"7f88f9a4a5c98b5531ff4cc21c29308bb949e8f4205e68e3061c8375c7e2f8a8"} Oct 30 20:03:20.829118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.828771 4959 scope.go:122] "RemoveContainer" containerID="1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911" Oct 30 20:03:20.842285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.842266 4959 scope.go:122] "RemoveContainer" containerID="1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911" Oct 30 20:03:20.842684 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:03:20.842657 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911\": container with ID starting with 1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911 not found: ID does not exist" containerID="1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911" Oct 30 20:03:20.842769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.842695 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911"} err="failed to get container status \"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911\": rpc error: code = NotFound desc = could not find container \"1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911\": container with ID starting with 1a45c25e83d0c2845d0d339d9eca6bb45408a851cce23c87735c9aa86b1cd911 not found: ID does not exist" Oct 30 20:03:20.927997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.927959 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 20:03:20.931754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:20.931723 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-plkw7"] Oct 30 20:03:21.920726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:21.920695 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.036526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:22.031513 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" path="/var/lib/kubelet/pods/ff97fb2a-8949-4354-a737-0b40e026b6b1/volumes" Oct 30 20:03:23.509290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:23.509258 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:25.191248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:25.191218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.787397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:26.787368 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.443951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:28.443914 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.111929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:30.111895 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.394530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:30.394485 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 20:03:30.394737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:30.394715 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" containerID="cri-o://a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537" gracePeriod=30 Oct 30 20:03:30.445031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:30.444998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.907375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:31.907348 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.957897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:33.957861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.225885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:34.225796 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.486453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.486411 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerName="main" containerID="cri-o://cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86" gracePeriod=45 Oct 30 20:03:35.525825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.525795 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.866344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.866064 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.881099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.881074 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 20:03:35.885647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.885628 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 20:03:35.903120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903093 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903226 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903292 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903320 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903379 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903405 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\" (UniqueName: \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.903441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903434 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903480 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.903648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903557 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.903648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903601 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.903648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903637 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.903857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903675 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.903857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903712 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.903857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903804 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\" (UniqueName: \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903835 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903881 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903934 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\") pod \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\" (UID: \"72d5fc9e-e7f6-49d9-8597-bc361ccdd177\") " Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903969 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.903978 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.904005 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\") pod \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\" (UID: \"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd\") " Oct 30 20:03:35.904176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.904059 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.904578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.904521 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:35.904730 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.904701 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905257 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905285 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905300 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905316 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905331 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905346 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905361 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.905396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.905374 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:35.906346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.906322 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "kube-api-access-b8vw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:35.906479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.906339 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:35.906938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.906889 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:35.907088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.907064 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.907363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.907348 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl" pod "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" (UID: "663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd"). InnerVolumeSpecName "kube-api-access-88dhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:35.907578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.907554 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:35.907949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.907931 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm" pod "72d5fc9e-e7f6-49d9-8597-bc361ccdd177" (UID: "72d5fc9e-e7f6-49d9-8597-bc361ccdd177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:35.980458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.980383 4959 generic.go:356] "Generic (PLEG): container finished" podID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerID="cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86" exitCode=0 Oct 30 20:03:35.980662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.980447 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" event={"ID":"72d5fc9e-e7f6-49d9-8597-bc361ccdd177","Type":"ContainerDied","Data":"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86"} Oct 30 20:03:35.980662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.980476 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" Oct 30 20:03:35.980662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.980541 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" event={"ID":"72d5fc9e-e7f6-49d9-8597-bc361ccdd177","Type":"ContainerDied","Data":"0adb890635d2807885b5260b95d3e9ee3b8700071836f5a13910b23a9265c726"} Oct 30 20:03:35.980662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.980568 4959 scope.go:122] "RemoveContainer" containerID="cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86" Oct 30 20:03:35.985075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.982524 4959 generic.go:356] "Generic (PLEG): container finished" podID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerID="a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537" exitCode=0 Oct 30 20:03:35.985075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.982554 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" event={"ID":"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd","Type":"ContainerDied","Data":"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537"} Oct 30 20:03:35.985075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.982617 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" Oct 30 20:03:35.985075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:35.982627 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7" event={"ID":"663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd","Type":"ContainerDied","Data":"5d80f0194a90fe6ccfb6f17a20cdcab106916e72e23ab13141dce32dcf5f9738"} Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008306 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008330 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vw5\" (UniqueName: \"kubernetes.io/projected/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-kube-api-access-b8vw5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008343 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dhl\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-kube-api-access-88dhl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008352 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008361 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d5fc9e-e7f6-49d9-8597-bc361ccdd177-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008368 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.014699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.008375 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:03:36.017616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.017595 4959 scope.go:122] "RemoveContainer" containerID="cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86" Oct 30 20:03:36.018030 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:03:36.018003 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86\": container with ID starting with cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86 not found: ID does not exist" containerID="cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86" Oct 30 20:03:36.018106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.018043 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86"} err="failed to get container status \"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86\": rpc error: code = NotFound desc = could not find container \"cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86\": container with ID starting with cda8300db46d6bdea7831704d73af60f79948abec6a8c7b0ea0bc1a4c834cb86 not found: ID does not exist" Oct 30 20:03:36.018106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.018067 4959 scope.go:122] "RemoveContainer" containerID="a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537" Oct 30 20:03:36.033908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.033889 4959 scope.go:122] "RemoveContainer" containerID="a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537" Oct 30 20:03:36.034237 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:03:36.034211 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537\": container with ID starting with a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537 not found: ID does not exist" containerID="a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537" Oct 30 20:03:36.034327 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.034248 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537"} err="failed to get container status \"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537\": rpc error: code = NotFound desc = could not find container \"a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537\": container with ID starting with a1c9cdbbc8b2cbe1651d8199ecc2d3924ae8ebf926e377eda038cbcf2dc34537 not found: ID does not exist" Oct 30 20:03:36.082978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.082951 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 20:03:36.089610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.089581 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8"] Oct 30 20:03:36.326905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.326820 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 20:03:36.341291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.341262 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-2svg7"] Oct 30 20:03:36.724532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:36.724457 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2224e496/e2e-lifecycle-model-2224e496-kserve-5564f484df-rj2k8" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:03:37.368986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:37.368950 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.002012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:38.001978 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" path="/var/lib/kubelet/pods/663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd/volumes" Oct 30 20:03:38.002613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:38.002529 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" path="/var/lib/kubelet/pods/72d5fc9e-e7f6-49d9-8597-bc361ccdd177/volumes" Oct 30 20:03:39.281932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:39.281902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.698149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.698112 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:03:40.699239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699218 4959 cpu_manager.go:414] "containerMap: removing container" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerName="main" Oct 30 20:03:40.699239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699240 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerName="main" Oct 30 20:03:40.699371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699312 4959 cpu_manager.go:414] "containerMap: removing container" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" Oct 30 20:03:40.699371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699318 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" Oct 30 20:03:40.699371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699338 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" Oct 30 20:03:40.699371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699343 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" Oct 30 20:03:40.699535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699506 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="663da9c7-6fc8-4c35-b9b1-cc77d2fecdcd" containerName="istio-proxy" Oct 30 20:03:40.699535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699517 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff97fb2a-8949-4354-a737-0b40e026b6b1" containerName="istio-proxy" Oct 30 20:03:40.699535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.699528 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d5fc9e-e7f6-49d9-8597-bc361ccdd177" containerName="main" Oct 30 20:03:40.706129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.706106 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.755079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.755045 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.853999 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854056 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzz8\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854105 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854167 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854299 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854442 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854476 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854611 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.870522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.854700 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.959588 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.959751 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.959788 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.959848 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.959922 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.960000 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.960029 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzz8\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.960070 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.960128 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.960663 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.961646 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.961885 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.962106 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.980566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.963303 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:40.996525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:40.990157 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:41.021745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.016574 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzz8\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:41.021745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.018183 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:41.024654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.023066 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:41.045839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.040951 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:41.448095 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:03:41.448062 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4585d42_af04_4dc2_8943_8ed6bf9a5b6c.slice/crio-999aaef5c3e2aa8e06edf211dcfda4b4f1fdeea892aab8900f7557d52d3c872e WatchSource:0}: Error finding container 999aaef5c3e2aa8e06edf211dcfda4b4f1fdeea892aab8900f7557d52d3c872e: Status 404 returned error can't find the container with id 999aaef5c3e2aa8e06edf211dcfda4b4f1fdeea892aab8900f7557d52d3c872e Oct 30 20:03:41.448728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.448660 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:03:41.451368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.451337 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:41.451470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.451432 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:41.451470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.451462 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:41.605422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:41.605383 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.272108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:42.272076 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" event={"ID":"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c","Type":"ContainerStarted","Data":"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0"} Oct 30 20:03:42.272649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:42.272131 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" event={"ID":"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c","Type":"ContainerStarted","Data":"999aaef5c3e2aa8e06edf211dcfda4b4f1fdeea892aab8900f7557d52d3c872e"} Oct 30 20:03:42.350191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:42.350137 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" podStartSLOduration=2.350122573 podStartE2EDuration="2.350122573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:42.332682233 +0000 UTC m=+1746.461916178" watchObservedRunningTime="2026-10-30 20:03:42.350122573 +0000 UTC m=+1746.479356493" Oct 30 20:03:43.042463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:43.042424 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:43.049411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:43.049376 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:43.336874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:43.336773 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:43.345530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:43.342872 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:03:44.000017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:43.999973 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.521650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:45.521619 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.784063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:45.783984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.894627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:45.894587 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.022435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:46.022405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.067115 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:03:46.067033 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:03:46.149382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:46.149347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.562582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:46.562541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.709168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:46.709133 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.229301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:47.229261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.346698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:47.346650 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.115766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:48.115716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.811203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:49.811169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.476758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:51.476716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.644187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:51.644157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.919565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:51.919531 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.142286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.142253 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.255045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.254972 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.291706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.291674 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:03:53.296526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.296484 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.325643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.325617 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-58d72c-dockercfg-dpwxg\"" Oct 30 20:03:53.331270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.331240 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:03:53.349961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.349931 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-58d72c\"" Oct 30 20:03:53.635771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.606828 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:03:53.640890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.640861 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.704930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.668616 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:03:53.705194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.684453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.705362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.698087 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.705739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.705713 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vlb\" (UniqueName: \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.705950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.705934 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cr8p\" (UniqueName: \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.706085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.706066 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.833517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.832969 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vlb\" (UniqueName: \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.833517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.833075 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cr8p\" (UniqueName: \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.833517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.833123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.833517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.833349 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.834398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.834371 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.858525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.852739 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.867694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.867663 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cr8p\" (UniqueName: \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\") pod \"payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:53.867896 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.867713 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vlb\" (UniqueName: \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\") pod \"payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:53.911815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:53.911728 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:54.056654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.056626 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:54.232041 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:03:54.232005 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6260a9_9d30_47fc_9348_5d2ca42b3bd4.slice/crio-f10b64cfb9511e8bb6e23fd13e12ac95a0066e6500f6916877484ca0c83116da WatchSource:0}: Error finding container f10b64cfb9511e8bb6e23fd13e12ac95a0066e6500f6916877484ca0c83116da: Status 404 returned error can't find the container with id f10b64cfb9511e8bb6e23fd13e12ac95a0066e6500f6916877484ca0c83116da Oct 30 20:03:54.233132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.233110 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:03:54.370148 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:03:54.370120 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a19396_fc1d_4247_bd55_b5e8f08d3623.slice/crio-2b36d97c87ec326810215561b26e5ae5bf41e05010feb95be62dff24bc6b423c WatchSource:0}: Error finding container 2b36d97c87ec326810215561b26e5ae5bf41e05010feb95be62dff24bc6b423c: Status 404 returned error can't find the container with id 2b36d97c87ec326810215561b26e5ae5bf41e05010feb95be62dff24bc6b423c Oct 30 20:03:54.371847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.371818 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:03:54.648763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.648724 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" event={"ID":"86a19396-fc1d-4247-bd55-b5e8f08d3623","Type":"ContainerStarted","Data":"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697"} Oct 30 20:03:54.648979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.648790 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" event={"ID":"86a19396-fc1d-4247-bd55-b5e8f08d3623","Type":"ContainerStarted","Data":"2b36d97c87ec326810215561b26e5ae5bf41e05010feb95be62dff24bc6b423c"} Oct 30 20:03:54.648979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.648816 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:03:54.650611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.650587 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" event={"ID":"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4","Type":"ContainerStarted","Data":"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a"} Oct 30 20:03:54.650733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.650626 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" event={"ID":"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4","Type":"ContainerStarted","Data":"f10b64cfb9511e8bb6e23fd13e12ac95a0066e6500f6916877484ca0c83116da"} Oct 30 20:03:54.650733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.650708 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:03:54.700346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.700286 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" podStartSLOduration=1.700273695 podStartE2EDuration="1.700273695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:54.698233379 +0000 UTC m=+1758.827467301" watchObservedRunningTime="2026-10-30 20:03:54.700273695 +0000 UTC m=+1758.829507615" Oct 30 20:03:54.744418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:54.744366 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" podStartSLOduration=1.744352546 podStartE2EDuration="1.744352546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:54.740675275 +0000 UTC m=+1758.869909197" watchObservedRunningTime="2026-10-30 20:03:54.744352546 +0000 UTC m=+1758.873586467" Oct 30 20:03:55.698785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.698748 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:03:55.704402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.704379 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.741625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.741593 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:03:55.818755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.818709 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.818958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.818762 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.818958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.818894 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819090 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819284 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819349 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819482 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819606 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ffw\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.819673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.819635 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921389 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921351 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921458 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ffw\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921487 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921563 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921594 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921644 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921776 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.921947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921904 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.921968 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.922011 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.922044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.922145 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.922259 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.922421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.922400 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.924180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.924159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.924268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.924254 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.943188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.943159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ffw\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.943457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.943436 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\") pod \"e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:55.956655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:55.956573 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.066178 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.066145 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:56.425228 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:03:56.425197 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f0cb74_8000_4637_97d4_894d21e11e8d.slice/crio-daa481468fb1e80061bbb1220cadcd2ac42347e23d9ab7aba3969a3447e5fc9f WatchSource:0}: Error finding container daa481468fb1e80061bbb1220cadcd2ac42347e23d9ab7aba3969a3447e5fc9f: Status 404 returned error can't find the container with id daa481468fb1e80061bbb1220cadcd2ac42347e23d9ab7aba3969a3447e5fc9f Oct 30 20:03:56.426894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.426866 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:03:56.428334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.428299 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:56.428402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.428368 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:56.428402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.428395 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:03:56.668999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.668957 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" event={"ID":"c5f0cb74-8000-4637-97d4-894d21e11e8d","Type":"ContainerStarted","Data":"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b"} Oct 30 20:03:56.669190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.669029 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" event={"ID":"c5f0cb74-8000-4637-97d4-894d21e11e8d","Type":"ContainerStarted","Data":"daa481468fb1e80061bbb1220cadcd2ac42347e23d9ab7aba3969a3447e5fc9f"} Oct 30 20:03:56.744388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:56.744274 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" podStartSLOduration=1.744255772 podStartE2EDuration="1.744255772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:56.742576777 +0000 UTC m=+1760.871810701" watchObservedRunningTime="2026-10-30 20:03:56.744255772 +0000 UTC m=+1760.873489690" Oct 30 20:03:57.074058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:57.073971 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:57.081195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:57.081163 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:57.718428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:57.718395 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:57.720350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:57.719821 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:03:58.527479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:58.527448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.467806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:59.467779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.588341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:59.588307 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.713694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:59.713431 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.905833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:59.905800 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 20:03:59.906091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:03:59.906056 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" containerID="cri-o://f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde" gracePeriod=30 Oct 30 20:04:00.034759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.034720 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 20:04:00.034993 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.034963 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" containerID="cri-o://09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768" gracePeriod=30 Oct 30 20:04:00.048427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.048394 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:00.124344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.124313 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 20:04:00.124596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.124569 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" containerID="cri-o://041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0" gracePeriod=30 Oct 30 20:04:00.609507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.609451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:00.883421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:00.883300 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.899095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:01.899058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.277943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:03.277910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.406094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.406065 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 20:04:05.447101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.447079 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 20:04:05.533139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533054 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533104 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533148 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.533396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533194 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.533396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533240 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533313 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.533396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533339 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533447 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533472 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533538 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533594 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533628 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.533742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533710 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533757 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533790 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533818 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533825 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.533949 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.534074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534040 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\") pod \"06bcb85a-4bc2-4156-ab40-4b180f654402\" (UID: \"06bcb85a-4bc2-4156-ab40-4b180f654402\") " Oct 30 20:04:05.534394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534080 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.534394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534084 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\") pod \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\" (UID: \"0d22d3b6-dfaa-4cfb-969f-e3950f569730\") " Oct 30 20:04:05.534394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534305 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.534584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534431 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.534584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534552 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.534584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534564 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.534858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534836 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:05.535076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.534926 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535195 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535673 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535689 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535698 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535707 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535717 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535725 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535742 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535749 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535757 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.535810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.535766 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06bcb85a-4bc2-4156-ab40-4b180f654402-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.537484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.537455 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.538320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.538272 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.539611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.539470 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:05.539687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.539635 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "kube-api-access-wljdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.540853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.540816 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "kube-api-access-txsbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.541575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.540954 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy" pod "06bcb85a-4bc2-4156-ab40-4b180f654402" (UID: "06bcb85a-4bc2-4156-ab40-4b180f654402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.541905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.541833 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.542437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.542323 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo" pod "0d22d3b6-dfaa-4cfb-969f-e3950f569730" (UID: "0d22d3b6-dfaa-4cfb-969f-e3950f569730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:05.550262 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.550235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.637660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637621 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637660 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637675 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljdk\" (UniqueName: \"kubernetes.io/projected/0d22d3b6-dfaa-4cfb-969f-e3950f569730-kube-api-access-wljdk\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637692 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsbz\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-kube-api-access-txsbz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637706 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637718 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d22d3b6-dfaa-4cfb-969f-e3950f569730-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637731 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.637930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.637743 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bcb85a-4bc2-4156-ab40-4b180f654402-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.661428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.661401 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:04:05.661618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.661484 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:04:05.662770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.662742 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.674403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.674379 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753272 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753461 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753530 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753620 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753888 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753931 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.753991 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.754023 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.758527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.754076 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\") pod \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\" (UID: \"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6\") " Oct 30 20:04:05.794573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.788614 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.814521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.801964 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.814521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.803521 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:05.814521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.806535 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:05.814521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.806760 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.838520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.832946 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858397 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858434 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858448 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858462 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858473 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.858485 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.877299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.881845 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.884248 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:05.895337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.894668 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw" pod "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" (UID: "3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6"). InnerVolumeSpecName "kube-api-access-mv5pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:05.964438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.963792 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.964438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.963822 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:05.964438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:05.963836 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5pw\" (UniqueName: \"kubernetes.io/projected/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6-kube-api-access-mv5pw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:06.044347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.044317 4959 generic.go:356] "Generic (PLEG): container finished" podID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerID="f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde" exitCode=0 Oct 30 20:04:06.044613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.044566 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.046104 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerDied","Data":"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde"} Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.046169 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx" event={"ID":"3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6","Type":"ContainerDied","Data":"6a4a0c4d6cc4064e2310eaca1e96eff085ec500fec3557efe70df6fbbf9572eb"} Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.046201 4959 scope.go:122] "RemoveContainer" containerID="f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde" Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.047813 4959 generic.go:356] "Generic (PLEG): container finished" podID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerID="041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0" exitCode=0 Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.048073 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" event={"ID":"06bcb85a-4bc2-4156-ab40-4b180f654402","Type":"ContainerDied","Data":"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0"} Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.048111 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" event={"ID":"06bcb85a-4bc2-4156-ab40-4b180f654402","Type":"ContainerDied","Data":"7a4caafd4adea65938afd886c6ef9ae8c2c530eebcbb655665cd95d93f288d79"} Oct 30 20:04:06.048975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.048176 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx" Oct 30 20:04:06.050720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.050644 4959 generic.go:356] "Generic (PLEG): container finished" podID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerID="09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768" exitCode=0 Oct 30 20:04:06.050924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.050896 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" event={"ID":"0d22d3b6-dfaa-4cfb-969f-e3950f569730","Type":"ContainerDied","Data":"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768"} Oct 30 20:04:06.051120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.051085 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" event={"ID":"0d22d3b6-dfaa-4cfb-969f-e3950f569730","Type":"ContainerDied","Data":"22fc3cd86096d6d7c66211035ecd9f25aa7543912b46534f52dace1e951fa549"} Oct 30 20:04:06.051360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.051343 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p" Oct 30 20:04:06.112647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.112535 4959 scope.go:122] "RemoveContainer" containerID="0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34" Oct 30 20:04:06.128154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.128130 4959 scope.go:122] "RemoveContainer" containerID="f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde" Oct 30 20:04:06.128619 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:06.128589 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde\": container with ID starting with f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde not found: ID does not exist" containerID="f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde" Oct 30 20:04:06.128734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.128632 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde"} err="failed to get container status \"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde\": rpc error: code = NotFound desc = could not find container \"f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde\": container with ID starting with f106a3dd8f79de7b9c2b305d6ffd4bed4e133962cfc3ae014a5ac6ce80f0afde not found: ID does not exist" Oct 30 20:04:06.128734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.128659 4959 scope.go:122] "RemoveContainer" containerID="0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34" Oct 30 20:04:06.128996 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:06.128972 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34\": container with ID starting with 0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34 not found: ID does not exist" containerID="0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34" Oct 30 20:04:06.129069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.129007 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34"} err="failed to get container status \"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34\": rpc error: code = NotFound desc = could not find container \"0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34\": container with ID starting with 0e612bc45a7b6d9ab2704995604d55298d88d5b571b231644882395a3aadac34 not found: ID does not exist" Oct 30 20:04:06.129069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.129031 4959 scope.go:122] "RemoveContainer" containerID="041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0" Oct 30 20:04:06.139579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.139557 4959 scope.go:122] "RemoveContainer" containerID="041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0" Oct 30 20:04:06.139898 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:06.139876 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0\": container with ID starting with 041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0 not found: ID does not exist" containerID="041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0" Oct 30 20:04:06.139960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.139906 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0"} err="failed to get container status \"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0\": rpc error: code = NotFound desc = could not find container \"041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0\": container with ID starting with 041656bdccaf198c75b96c50b1580b444309c61538ad1f01303dfb706e9970c0 not found: ID does not exist" Oct 30 20:04:06.139960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.139924 4959 scope.go:122] "RemoveContainer" containerID="09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768" Oct 30 20:04:06.150871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.150850 4959 scope.go:122] "RemoveContainer" containerID="09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768" Oct 30 20:04:06.151177 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:06.151156 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768\": container with ID starting with 09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768 not found: ID does not exist" containerID="09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768" Oct 30 20:04:06.151259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.151191 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768"} err="failed to get container status \"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768\": rpc error: code = NotFound desc = could not find container \"09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768\": container with ID starting with 09b487ef3529408eed4881859a8da07d43c8223387a8d31d1bfeaaea05f21768 not found: ID does not exist" Oct 30 20:04:06.168133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.168105 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 20:04:06.173994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.173967 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-b58lx"] Oct 30 20:04:06.211560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.211532 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 20:04:06.215518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.215477 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-6mwpx"] Oct 30 20:04:06.254464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.254437 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 20:04:06.259087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.259066 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bpr7p"] Oct 30 20:04:06.720843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:06.720813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.204840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.204807 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.317879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.317826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.328887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.328862 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:04:07.329910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329871 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.329910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329895 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.329910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329914 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" Oct 30 20:04:07.330152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329919 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" Oct 30 20:04:07.330152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329930 4959 cpu_manager.go:414] "containerMap: removing container" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" Oct 30 20:04:07.330152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329934 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" Oct 30 20:04:07.330152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329961 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.330152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.329966 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.330454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.330193 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.330454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.330213 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" containerName="istio-proxy" Oct 30 20:04:07.330454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.330224 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" containerName="istio-proxy" Oct 30 20:04:07.330454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.330239 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" containerName="istio-proxy" Oct 30 20:04:07.336247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.336229 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.359403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.359371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:04:07.394652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.394620 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:04:07.399811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.399786 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.419968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.419540 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6cd2ad-dockercfg-rwvrf\"" Oct 30 20:04:07.423819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.423798 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6cd2ad\"" Oct 30 20:04:07.425009 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.424987 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:04:07.604576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.591264 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54cws\" (UniqueName: \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.604576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.591404 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jwl\" (UniqueName: \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.604576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.591451 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.604576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.591478 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.713540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.693658 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.763646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.761290 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.763646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.761791 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54cws\" (UniqueName: \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.763646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.761969 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jwl\" (UniqueName: \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.763646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.739516 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.763646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.763284 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.787091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.785685 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jwl\" (UniqueName: \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\") pod \"payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:07.788329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.788308 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54cws\" (UniqueName: \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\") pod \"payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:07.954453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:07.954420 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:08.003386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.003340 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bcb85a-4bc2-4156-ab40-4b180f654402" path="/var/lib/kubelet/pods/06bcb85a-4bc2-4156-ab40-4b180f654402/volumes" Oct 30 20:04:08.004119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.004094 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d22d3b6-dfaa-4cfb-969f-e3950f569730" path="/var/lib/kubelet/pods/0d22d3b6-dfaa-4cfb-969f-e3950f569730/volumes" Oct 30 20:04:08.004726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.004705 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6" path="/var/lib/kubelet/pods/3a98ac8a-04b4-4aac-8b5d-a366e28c0ba6/volumes" Oct 30 20:04:08.020352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.020325 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:08.122840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.122794 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.295042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.295015 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:04:08.295676 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:04:08.295646 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac7786e_fbd5_4d81_af38_1b336196bd05.slice/crio-bbf8ca110dbfb0a2ef5b0e3c7271adde149c0bac3cdc7dd0017286a0e48c4010 WatchSource:0}: Error finding container bbf8ca110dbfb0a2ef5b0e3c7271adde149c0bac3cdc7dd0017286a0e48c4010: Status 404 returned error can't find the container with id bbf8ca110dbfb0a2ef5b0e3c7271adde149c0bac3cdc7dd0017286a0e48c4010 Oct 30 20:04:08.354872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.354846 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:04:08.354980 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:04:08.354886 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c04304_9a22_469b_8152_593ddecb4254.slice/crio-2e6e79b9385b92d3f1ad470ccc52d26c2b6fdef1e4a2cc584c52b1ec21545f01 WatchSource:0}: Error finding container 2e6e79b9385b92d3f1ad470ccc52d26c2b6fdef1e4a2cc584c52b1ec21545f01: Status 404 returned error can't find the container with id 2e6e79b9385b92d3f1ad470ccc52d26c2b6fdef1e4a2cc584c52b1ec21545f01 Oct 30 20:04:08.798631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:08.798588 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.082868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.082785 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" event={"ID":"70c04304-9a22-469b-8152-593ddecb4254","Type":"ContainerStarted","Data":"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211"} Oct 30 20:04:09.082868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.082836 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" event={"ID":"70c04304-9a22-469b-8152-593ddecb4254","Type":"ContainerStarted","Data":"2e6e79b9385b92d3f1ad470ccc52d26c2b6fdef1e4a2cc584c52b1ec21545f01"} Oct 30 20:04:09.082868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.082866 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:09.084510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.084469 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" event={"ID":"2ac7786e-fbd5-4d81-af38-1b336196bd05","Type":"ContainerStarted","Data":"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade"} Oct 30 20:04:09.084659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.084545 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" event={"ID":"2ac7786e-fbd5-4d81-af38-1b336196bd05","Type":"ContainerStarted","Data":"bbf8ca110dbfb0a2ef5b0e3c7271adde149c0bac3cdc7dd0017286a0e48c4010"} Oct 30 20:04:09.084659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.084568 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:09.140017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.139971 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" podStartSLOduration=2.139957819 podStartE2EDuration="2.139957819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:09.138588344 +0000 UTC m=+1773.267822264" watchObservedRunningTime="2026-10-30 20:04:09.139957819 +0000 UTC m=+1773.269191739" Oct 30 20:04:09.186338 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.186284 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" podStartSLOduration=2.186269639 podStartE2EDuration="2.186269639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:09.18270158 +0000 UTC m=+1773.311935500" watchObservedRunningTime="2026-10-30 20:04:09.186269639 +0000 UTC m=+1773.315503560" Oct 30 20:04:09.264815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.264789 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.696605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.696575 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:04:09.715967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.715938 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.734557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.732685 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:04:09.757543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.757509 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.780335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.780303 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jskp\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.780556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.780530 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.780697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.780675 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.780816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.780794 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.780850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.780834 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.781028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.781010 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.781067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.781047 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.781193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.781177 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.781230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.781205 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.787863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.787838 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4a31d3-openshift-default-dockercfg-mnpnt\"" Oct 30 20:04:09.871320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.871285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.882350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882325 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882357 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882420 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882443 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882536 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jskp\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882651 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882732 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882805 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882838 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.882917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.882899 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.883160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.883138 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.883277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.883227 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.883529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.883507 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.885115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.885092 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.885336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.885317 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.905882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.905858 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.906033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.906008 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jskp\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\") pod \"e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:09.973827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:09.973736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.047574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.047545 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:10.078029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.078002 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.769541 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:04:10.767603 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode430d411_2526_4b02_8452_d9eb1fbe99fb.slice/crio-a007d54a7a12a92c4077dbf829d1e92bf7e678398f29f797602c2af9c12144b1 WatchSource:0}: Error finding container a007d54a7a12a92c4077dbf829d1e92bf7e678398f29f797602c2af9c12144b1: Status 404 returned error can't find the container with id a007d54a7a12a92c4077dbf829d1e92bf7e678398f29f797602c2af9c12144b1 Oct 30 20:04:10.769541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.769069 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:04:10.774077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.774042 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:04:10.774188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.774114 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:04:10.774188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:10.774142 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:04:11.111777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:11.111678 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerStarted","Data":"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780"} Oct 30 20:04:11.111777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:11.111742 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerStarted","Data":"a007d54a7a12a92c4077dbf829d1e92bf7e678398f29f797602c2af9c12144b1"} Oct 30 20:04:11.938744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:11.938710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.049516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:12.048862 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:12.070551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:12.068860 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:12.120254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:12.120227 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:12.121383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:12.121358 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:04:12.145611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:12.145564 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podStartSLOduration=3.145550684 podStartE2EDuration="3.145550684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:11.173916666 +0000 UTC m=+1775.303150591" watchObservedRunningTime="2026-10-30 20:04:12.145550684 +0000 UTC m=+1776.274784606" Oct 30 20:04:13.710766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:13.710728 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.823764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:13.823721 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.950683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:13.950654 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.296548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.286606 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:04:14.296548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.286889 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" podUID="c5f0cb74-8000-4637-97d4-894d21e11e8d" containerName="istio-proxy" containerID="cri-o://3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b" gracePeriod=30 Oct 30 20:04:14.302484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.302435 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.304174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.304157 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 20:04:14.304605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.304581 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" containerID="cri-o://0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211" gracePeriod=30 Oct 30 20:04:14.311190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.311163 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:04:14.311435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.311382 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" podUID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" containerName="istio-proxy" containerID="cri-o://bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0" gracePeriod=30 Oct 30 20:04:14.670512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.670464 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 20:04:14.670784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:14.670726 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" containerID="cri-o://1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635" gracePeriod=30 Oct 30 20:04:15.200045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.200012 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 20:04:15.200596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.200228 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" containerID="cri-o://521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72" gracePeriod=30 Oct 30 20:04:15.625658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.625567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.744946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.744905 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.800510 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.800470 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:04:15.807540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.807478 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:15.823828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.823801 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4a31d3\"" Oct 30 20:04:15.824131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.824112 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4a31d3-dockercfg-qm8bm\"" Oct 30 20:04:15.824837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:15.824815 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:04:16.075524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.075328 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.075524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.075463 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6td\" (UniqueName: \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.169521 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:16.169465 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:04:16.209881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.209841 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6td\" (UniqueName: \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.210285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.210149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.210903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.210882 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.232599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.232571 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6td\" (UniqueName: \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\") pod \"payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.426260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.426232 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:16.672811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.672783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:16.802436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:16.802398 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:04:17.222205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:17.222165 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" event={"ID":"36d7a7bc-f28b-47a0-a313-18cbe1d80262","Type":"ContainerStarted","Data":"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e"} Oct 30 20:04:17.222721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:17.222242 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" event={"ID":"36d7a7bc-f28b-47a0-a313-18cbe1d80262","Type":"ContainerStarted","Data":"b8ac53c30cfe34ebed0328a28c748e588b5e8c6fcbb9b3dc5006ac6a1b5423e5"} Oct 30 20:04:17.223612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:17.223588 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:17.270207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:17.270151 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" podStartSLOduration=2.270136201 podStartE2EDuration="2.270136201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:17.268764467 +0000 UTC m=+1781.397998389" watchObservedRunningTime="2026-10-30 20:04:17.270136201 +0000 UTC m=+1781.399370121" Oct 30 20:04:18.309339 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:18.309306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.296889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.296858 4959 patch_prober.go:29] interesting pod/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:04:19.297058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.296908 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:04:19.638806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.638768 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:04:19.714167 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.711443 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 20:04:19.760066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.759704 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767640 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767690 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767741 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767854 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767884 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767925 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.767959 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768017 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768138 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768186 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768216 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768284 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768313 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768411 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768541 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768606 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768656 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\") pod \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\" (UID: \"a162056a-76a0-4ea2-9e13-6e7b28623c8b\") " Oct 30 20:04:19.769985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.768701 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\") pod \"c5f0cb74-8000-4637-97d4-894d21e11e8d\" (UID: \"c5f0cb74-8000-4637-97d4-894d21e11e8d\") " Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772003 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772114 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772195 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772416 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772424 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772340 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.772871 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.773402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.773164 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.777454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.775930 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.777624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.777580 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.779968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.779938 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.780079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.779977 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.780079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.780029 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.780079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.780071 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.780745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.780716 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw" pod "c5f0cb74-8000-4637-97d4-894d21e11e8d" (UID: "c5f0cb74-8000-4637-97d4-894d21e11e8d"). InnerVolumeSpecName "kube-api-access-l7ffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.781417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.781391 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.781994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.781975 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "kube-api-access-84nsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.782241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.782215 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token" pod "a162056a-76a0-4ea2-9e13-6e7b28623c8b" (UID: "a162056a-76a0-4ea2-9e13-6e7b28623c8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.871008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.870918 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.870986 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871101 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871419 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871522 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871300 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.871807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871634 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.871807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871332 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.871807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871769 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.871807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871819 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.872142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871909 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.872142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.871952 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\") pod \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\" (UID: \"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c\") " Oct 30 20:04:19.872601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.872575 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.873012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.872989 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:19.873771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873742 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873793 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873811 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873826 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873837 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873851 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.873874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873864 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873880 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873895 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873909 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873922 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873936 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873952 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f0cb74-8000-4637-97d4-894d21e11e8d-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873964 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873978 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ffw\" (UniqueName: \"kubernetes.io/projected/c5f0cb74-8000-4637-97d4-894d21e11e8d-kube-api-access-l7ffw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873980 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.873994 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nsx\" (UniqueName: \"kubernetes.io/projected/a162056a-76a0-4ea2-9e13-6e7b28623c8b-kube-api-access-84nsx\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874010 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874026 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874038 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874052 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a162056a-76a0-4ea2-9e13-6e7b28623c8b-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874065 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874081 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a162056a-76a0-4ea2-9e13-6e7b28623c8b-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.874111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.874094 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f0cb74-8000-4637-97d4-894d21e11e8d-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:19.876098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.876070 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:19.877236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.877212 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:19.878325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.878303 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8" pod "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" (UID: "d4585d42-af04-4dc2-8943-8ed6bf9a5b6c"). InnerVolumeSpecName "kube-api-access-dmzz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:19.903181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.903147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.916082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.916052 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.985691 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986031 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986109 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986183 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986270 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986403 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986453 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986526 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:19.989522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:19.986684 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\") pod \"75ee3675-99d4-4e43-bb3b-b63c04424db1\" (UID: \"75ee3675-99d4-4e43-bb3b-b63c04424db1\") " Oct 30 20:04:20.006521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.003158 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.006521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.003192 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.006521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.003206 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.006521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.003222 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzz8\" (UniqueName: \"kubernetes.io/projected/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c-kube-api-access-dmzz8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.008284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.007932 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.026215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.026168 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.030541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.027761 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.030541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.027937 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.030541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.028397 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:20.076894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.064650 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:20.102658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.101625 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "kube-api-access-7nxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104918 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104940 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104951 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75ee3675-99d4-4e43-bb3b-b63c04424db1-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104962 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nxqn\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-kube-api-access-7nxqn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104970 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104980 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.105139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.104987 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.107140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.107116 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:04:20.107633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.107613 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:04:20.109033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.109005 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:20.117952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.117929 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy" pod "75ee3675-99d4-4e43-bb3b-b63c04424db1" (UID: "75ee3675-99d4-4e43-bb3b-b63c04424db1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.207230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.207194 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.207440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.207426 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75ee3675-99d4-4e43-bb3b-b63c04424db1-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.250353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.250320 4959 generic.go:356] "Generic (PLEG): container finished" podID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerID="0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211" exitCode=0 Oct 30 20:04:20.250561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.250377 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" Oct 30 20:04:20.250561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.250402 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" event={"ID":"a162056a-76a0-4ea2-9e13-6e7b28623c8b","Type":"ContainerDied","Data":"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211"} Oct 30 20:04:20.250561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.250463 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4" event={"ID":"a162056a-76a0-4ea2-9e13-6e7b28623c8b","Type":"ContainerDied","Data":"24996b9a43699dd222524b19c569f4f03fe0b80d47b89f5abaaf22fbc90fab92"} Oct 30 20:04:20.250561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.250510 4959 scope.go:122] "RemoveContainer" containerID="0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211" Oct 30 20:04:20.252390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.252363 4959 generic.go:356] "Generic (PLEG): container finished" podID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" containerID="bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0" exitCode=0 Oct 30 20:04:20.252577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.252428 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" Oct 30 20:04:20.252577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.252450 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" event={"ID":"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c","Type":"ContainerDied","Data":"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0"} Oct 30 20:04:20.252718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.252586 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg" event={"ID":"d4585d42-af04-4dc2-8943-8ed6bf9a5b6c","Type":"ContainerDied","Data":"999aaef5c3e2aa8e06edf211dcfda4b4f1fdeea892aab8900f7557d52d3c872e"} Oct 30 20:04:20.254232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.254209 4959 generic.go:356] "Generic (PLEG): container finished" podID="c5f0cb74-8000-4637-97d4-894d21e11e8d" containerID="3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b" exitCode=0 Oct 30 20:04:20.254317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.254248 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" Oct 30 20:04:20.254317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.254281 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" event={"ID":"c5f0cb74-8000-4637-97d4-894d21e11e8d","Type":"ContainerDied","Data":"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b"} Oct 30 20:04:20.254418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.254323 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89" event={"ID":"c5f0cb74-8000-4637-97d4-894d21e11e8d","Type":"ContainerDied","Data":"daa481468fb1e80061bbb1220cadcd2ac42347e23d9ab7aba3969a3447e5fc9f"} Oct 30 20:04:20.255982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.255953 4959 generic.go:356] "Generic (PLEG): container finished" podID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerID="1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635" exitCode=0 Oct 30 20:04:20.256078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.256062 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" event={"ID":"75ee3675-99d4-4e43-bb3b-b63c04424db1","Type":"ContainerDied","Data":"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635"} Oct 30 20:04:20.256136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.256091 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" event={"ID":"75ee3675-99d4-4e43-bb3b-b63c04424db1","Type":"ContainerDied","Data":"5550df82de81c4def46cc905956ede2ff6d8f5ff1d5502d2898d5e5702ffdae8"} Oct 30 20:04:20.256176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.256143 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4" Oct 30 20:04:20.323194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.323172 4959 scope.go:122] "RemoveContainer" containerID="0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211" Oct 30 20:04:20.323580 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:20.323551 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211\": container with ID starting with 0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211 not found: ID does not exist" containerID="0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211" Oct 30 20:04:20.323672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.323595 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211"} err="failed to get container status \"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211\": rpc error: code = NotFound desc = could not find container \"0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211\": container with ID starting with 0bfa1dc21432956e67484be3e1c7481bc21206e075d7b0b647f3105210518211 not found: ID does not exist" Oct 30 20:04:20.323672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.323622 4959 scope.go:122] "RemoveContainer" containerID="bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0" Oct 30 20:04:20.359414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.359382 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 20:04:20.370069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.368924 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-sprt4"] Oct 30 20:04:20.382594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.382092 4959 scope.go:122] "RemoveContainer" containerID="bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0" Oct 30 20:04:20.382773 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:20.382628 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0\": container with ID starting with bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0 not found: ID does not exist" containerID="bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0" Oct 30 20:04:20.382773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.382671 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0"} err="failed to get container status \"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0\": rpc error: code = NotFound desc = could not find container \"bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0\": container with ID starting with bf8dd7af360d01b94602f83a9030d13c0335ddf771e3fb6952801644eb3b87e0 not found: ID does not exist" Oct 30 20:04:20.382773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.382698 4959 scope.go:122] "RemoveContainer" containerID="3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b" Oct 30 20:04:20.400555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.400479 4959 scope.go:122] "RemoveContainer" containerID="3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b" Oct 30 20:04:20.400905 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:20.400878 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b\": container with ID starting with 3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b not found: ID does not exist" containerID="3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b" Oct 30 20:04:20.401017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.400916 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b"} err="failed to get container status \"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b\": rpc error: code = NotFound desc = could not find container \"3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b\": container with ID starting with 3d38335e5cccfdfe38935c8d617c5fda12a40939eae9c9f24a56de7121e1432b not found: ID does not exist" Oct 30 20:04:20.401017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.400942 4959 scope.go:122] "RemoveContainer" containerID="1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635" Oct 30 20:04:20.402516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.402455 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 20:04:20.406642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.406612 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-qstr4"] Oct 30 20:04:20.414837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.414814 4959 scope.go:122] "RemoveContainer" containerID="1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635" Oct 30 20:04:20.415182 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:20.415153 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635\": container with ID starting with 1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635 not found: ID does not exist" containerID="1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635" Oct 30 20:04:20.415274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.415191 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635"} err="failed to get container status \"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635\": rpc error: code = NotFound desc = could not find container \"1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635\": container with ID starting with 1e66730d33e66b08d429ca70523229097f7740a7b820de60222a9da18e3ba635 not found: ID does not exist" Oct 30 20:04:20.452996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.452956 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:04:20.458782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.458707 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-c5ptg"] Oct 30 20:04:20.497155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.497124 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:04:20.501544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.501522 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2224e496-openshift-default-759b9d9c85-n6b89"] Oct 30 20:04:20.544069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.544041 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 20:04:20.615061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615130 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615194 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615224 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615261 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615356 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615393 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615590 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.615723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.615704 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\") pod \"90eb72b2-1541-4c2c-8e15-48a27a19752b\" (UID: \"90eb72b2-1541-4c2c-8e15-48a27a19752b\") " Oct 30 20:04:20.616774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.616337 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:20.616774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.616605 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.616774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.616736 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eb72b2-1541-4c2c-8e15-48a27a19752b-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.617136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.616950 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.617248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.617129 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.617588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.617564 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.618401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.618379 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:20.619924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.619897 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:20.620014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.619947 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "kube-api-access-f5hcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:20.620054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.620024 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token" pod "90eb72b2-1541-4c2c-8e15-48a27a19752b" (UID: "90eb72b2-1541-4c2c-8e15-48a27a19752b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:20.719417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719334 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.719417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719370 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.719417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719384 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.719417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719398 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.719417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719411 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.720288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719426 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.720288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719439 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eb72b2-1541-4c2c-8e15-48a27a19752b-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:20.720288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:20.719451 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5hcl\" (UniqueName: \"kubernetes.io/projected/90eb72b2-1541-4c2c-8e15-48a27a19752b-kube-api-access-f5hcl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:04:21.266425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.266389 4959 generic.go:356] "Generic (PLEG): container finished" podID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerID="521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72" exitCode=0 Oct 30 20:04:21.266767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.266441 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" Oct 30 20:04:21.266767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.266469 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" event={"ID":"90eb72b2-1541-4c2c-8e15-48a27a19752b","Type":"ContainerDied","Data":"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72"} Oct 30 20:04:21.266767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.266560 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr" event={"ID":"90eb72b2-1541-4c2c-8e15-48a27a19752b","Type":"ContainerDied","Data":"beee01306c3a8e28c7886859b9cfe6e152696745ccdccfe968c28fb41658b700"} Oct 30 20:04:21.266767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.266594 4959 scope.go:122] "RemoveContainer" containerID="521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72" Oct 30 20:04:21.279418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.279395 4959 scope.go:122] "RemoveContainer" containerID="521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72" Oct 30 20:04:21.279793 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:21.279766 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72\": container with ID starting with 521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72 not found: ID does not exist" containerID="521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72" Oct 30 20:04:21.279891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.279804 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72"} err="failed to get container status \"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72\": rpc error: code = NotFound desc = could not find container \"521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72\": container with ID starting with 521fe47c69f081a8948de734f5bf14072a5ec44b90e20f8614ab6b2bca514b72 not found: ID does not exist" Oct 30 20:04:21.337663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.337625 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 20:04:21.350429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.350400 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-b4glr"] Oct 30 20:04:21.596474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:21.596385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.043517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:22.042608 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" path="/var/lib/kubelet/pods/75ee3675-99d4-4e43-bb3b-b63c04424db1/volumes" Oct 30 20:04:22.044053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:22.043536 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" path="/var/lib/kubelet/pods/90eb72b2-1541-4c2c-8e15-48a27a19752b/volumes" Oct 30 20:04:22.044237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:22.044207 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" path="/var/lib/kubelet/pods/a162056a-76a0-4ea2-9e13-6e7b28623c8b/volumes" Oct 30 20:04:22.053587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:22.049283 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f0cb74-8000-4637-97d4-894d21e11e8d" path="/var/lib/kubelet/pods/c5f0cb74-8000-4637-97d4-894d21e11e8d/volumes" Oct 30 20:04:22.053587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:22.050125 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" path="/var/lib/kubelet/pods/d4585d42-af04-4dc2-8943-8ed6bf9a5b6c/volumes" Oct 30 20:04:23.365177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:23.365142 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:25.137179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:25.137138 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:27.035995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:27.035958 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:28.233973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:28.233946 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:04:29.046609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:29.046579 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.616635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:30.616595 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.909815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:30.909787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.643556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:32.643486 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.067191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:34.067155 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.162545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:35.162517 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.450624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:35.450481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.707336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:35.707250 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.247626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:36.247602 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:04:36.257231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:36.257211 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:04:37.318062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:37.318022 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.375794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:39.375756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.492387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:39.492342 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.600047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:39.598669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.887051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:41.887019 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.063152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:42.063121 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.173334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:42.173219 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.424784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:42.424700 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.239533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:43.239479 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.747701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:43.747664 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:44.544860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:44.544830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:45.000846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:45.000808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.067112 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:04:46.067073 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:04:46.324868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:46.324786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.431481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:46.431451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.965792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:46.965760 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:47.888467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:47.888437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.003632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:48.003596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.868849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:48.868798 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.571443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.571408 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.679792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.679762 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.748234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.748200 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:04:49.749008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.748993 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c5f0cb74-8000-4637-97d4-894d21e11e8d" containerName="istio-proxy" Oct 30 20:04:49.749057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749010 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f0cb74-8000-4637-97d4-894d21e11e8d" containerName="istio-proxy" Oct 30 20:04:49.749057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749036 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" Oct 30 20:04:49.749057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749041 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749080 4959 cpu_manager.go:414] "containerMap: removing container" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749088 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749101 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749106 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749118 4959 cpu_manager.go:414] "containerMap: removing container" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" Oct 30 20:04:49.749163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749122 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" Oct 30 20:04:49.749348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749231 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ee3675-99d4-4e43-bb3b-b63c04424db1" containerName="istio-proxy" Oct 30 20:04:49.749348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749241 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4585d42-af04-4dc2-8943-8ed6bf9a5b6c" containerName="istio-proxy" Oct 30 20:04:49.749348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749250 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="90eb72b2-1541-4c2c-8e15-48a27a19752b" containerName="istio-proxy" Oct 30 20:04:49.749348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749258 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f0cb74-8000-4637-97d4-894d21e11e8d" containerName="istio-proxy" Oct 30 20:04:49.749348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.749268 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a162056a-76a0-4ea2-9e13-6e7b28623c8b" containerName="istio-proxy" Oct 30 20:04:49.754915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.754898 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:49.773064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.773042 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:04:49.782442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.782410 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c8fc40-dockercfg-c4bs9\"" Oct 30 20:04:49.782842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:49.782821 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c8fc40\"" Oct 30 20:04:50.008530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.000110 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.066964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.066859 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zj7\" (UniqueName: \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.168557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.168520 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.168750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.168727 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zj7\" (UniqueName: \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.169324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.169302 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.189973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.189942 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zj7\" (UniqueName: \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\") pod \"payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.370632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.370550 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:50.676397 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:04:50.676371 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb925d0cb_a3cd_452e_9742_07d79a2928d0.slice/crio-1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc WatchSource:0}: Error finding container 1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc: Status 404 returned error can't find the container with id 1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc Oct 30 20:04:50.679107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.679083 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:04:50.890997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:50.890962 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:51.689474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:51.689442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" event={"ID":"b925d0cb-a3cd-452e-9742-07d79a2928d0","Type":"ContainerStarted","Data":"9630128a3abba7ed84237e04425804d51c511706f4863ea82f2f1fbb8902f7f3"} Oct 30 20:04:51.689906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:51.689516 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" event={"ID":"b925d0cb-a3cd-452e-9742-07d79a2928d0","Type":"ContainerStarted","Data":"1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc"} Oct 30 20:04:51.689906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:51.689548 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:04:51.738700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:51.738649 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" podStartSLOduration=2.7386368169999997 podStartE2EDuration="2.738636817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:51.734768238 +0000 UTC m=+1815.864002159" watchObservedRunningTime="2026-10-30 20:04:51.738636817 +0000 UTC m=+1815.867870768" Oct 30 20:04:52.678317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:52.678286 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.140056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.140023 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:04:54.145244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.145221 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.166518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.165204 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6cd2ad-serving-cert\"" Oct 30 20:04:54.169634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.169606 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:04:54.226701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.226664 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.241290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.241260 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.241417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.241343 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.241474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.241459 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.241568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.241553 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmf2\" (UniqueName: \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.332526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.332465 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.342863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.342841 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.342989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.342917 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.342989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.342984 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.343107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.343031 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmf2\" (UniqueName: \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.343675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.343597 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.345626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.345608 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.345717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.345674 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.392427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.392158 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmf2\" (UniqueName: \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\") pod \"maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.457915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.457866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.467181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.467151 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:54.782235 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:04:54.782208 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6deb66_12d8_46ad_b87c_4e24fba3a487.slice/crio-ad1ce0847b9ef08a746d44deb5a9a6c042a743fdc52623fbe1a01039837ee266 WatchSource:0}: Error finding container ad1ce0847b9ef08a746d44deb5a9a6c042a743fdc52623fbe1a01039837ee266: Status 404 returned error can't find the container with id ad1ce0847b9ef08a746d44deb5a9a6c042a743fdc52623fbe1a01039837ee266 Oct 30 20:04:54.783777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.783756 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:04:54.783777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:54.783764 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:04:55.721553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:55.721486 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" event={"ID":"1b6deb66-12d8-46ad-b87c-4e24fba3a487","Type":"ContainerStarted","Data":"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1"} Oct 30 20:04:55.721964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:55.721574 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:04:55.721964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:55.721596 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" event={"ID":"1b6deb66-12d8-46ad-b87c-4e24fba3a487","Type":"ContainerStarted","Data":"ad1ce0847b9ef08a746d44deb5a9a6c042a743fdc52623fbe1a01039837ee266"} Oct 30 20:04:55.771661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:55.771607 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" podStartSLOduration=1.418923243 podStartE2EDuration="1.771592527s" totalImagesPullingTime="352.669284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:55.771082769 +0000 UTC m=+1819.900316691" watchObservedRunningTime="2026-10-30 20:04:55.771592527 +0000 UTC m=+1819.900826447" Oct 30 20:04:55.865590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:55.865552 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:56.493251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:56.493222 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:57.406783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:57.406746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.951934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:04:58.951902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.489217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:00.489180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:01.735162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:01.735132 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:05:02.092524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.092428 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:05:02.097399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.097378 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.113873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.113848 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4a31d3-serving-cert\"" Oct 30 20:05:02.120143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.120117 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:05:02.171983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.171945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.188307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.188277 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.188464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.188320 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.188532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.188465 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92jv\" (UniqueName: \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.188686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.188669 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.286597 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.286558 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.304230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.304187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.305038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.305011 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.305324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.305307 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92jv\" (UniqueName: \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.305675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.305652 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.337334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.312273 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.337334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.326724 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.338371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.338338 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.358353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.358275 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92jv\" (UniqueName: \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\") pod \"maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.393663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.393627 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.417403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.417375 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:02.700035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.700008 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:05:02.736515 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:05:02.736469 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cb5480_96e2_4a7e_8c61_78fb5e3431c0.slice/crio-449cb98baf7d3cc95d7474c2142bb9c198144980129c25bf376f9d9b7fe9f627 WatchSource:0}: Error finding container 449cb98baf7d3cc95d7474c2142bb9c198144980129c25bf376f9d9b7fe9f627: Status 404 returned error can't find the container with id 449cb98baf7d3cc95d7474c2142bb9c198144980129c25bf376f9d9b7fe9f627 Oct 30 20:05:02.738963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.738944 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:05:02.790317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:02.790279 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" event={"ID":"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0","Type":"ContainerStarted","Data":"449cb98baf7d3cc95d7474c2142bb9c198144980129c25bf376f9d9b7fe9f627"} Oct 30 20:05:03.803202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:03.803160 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" event={"ID":"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0","Type":"ContainerStarted","Data":"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560"} Oct 30 20:05:03.803768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:03.803367 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:03.847988 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:03.847934 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" podStartSLOduration=1.458326062 podStartE2EDuration="1.847920492s" totalImagesPullingTime="389.59443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.846000199 +0000 UTC m=+1827.975234121" watchObservedRunningTime="2026-10-30 20:05:03.847920492 +0000 UTC m=+1827.977154413" Oct 30 20:05:03.938673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:03.938636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:05.505721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:05.505686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.713746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:06.713696 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.818195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:06.818157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:07.017583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:07.017475 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.763721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:08.763682 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.824740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:09.824712 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:05:10.350275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:10.350229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.276915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:12.276877 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.707430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:14.707357 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:16.163520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.148071 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.252912 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:05:16.252875 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:05:16.259791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.258587 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.325814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.324580 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:16.370615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.370574 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Oct 30 20:05:16.404612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.404567 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded" Oct 30 20:05:16.892426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:16.892391 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:19.157362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:19.157327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:21.360387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:21.360341 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.109777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:23.109740 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.778346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:24.778310 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:26.688603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:26.688563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.969843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:27.969806 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.077985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:28.077951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.422438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:28.422403 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.681098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:28.681005 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.792588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:28.792478 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.149299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:29.149253 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.667621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:30.667582 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.787695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:30.787660 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.170722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:31.170684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.819757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:32.819716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.025333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:33.025293 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.132926 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:33.132901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.701473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:33.701441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.603834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:34.603800 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.709306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:34.709268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.821425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:34.821378 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.631456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:36.631419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.918670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:36.918637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:38.706607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:38.706573 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.031250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:40.031205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.619781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:40.619713 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.373944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:42.373913 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:44.184399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:44.184355 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:46.067255 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:05:46.067208 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:05:46.197691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:46.197646 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.528639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:47.528591 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.260019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:48.259957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:50.222027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:50.221993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.073281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:52.073234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.991079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:53.991045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:55.730147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:55.730108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:57.663512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:57.663458 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:59.291950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:05:59.291912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:01.010143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:01.010109 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:01.177470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:01.177427 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.915517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:02.914179 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.500879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:04.500817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:06.382836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:06.382781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:07.976309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:07.976257 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.620720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.620688 4959 generic.go:356] "Generic (PLEG): container finished" podID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerID="dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780" exitCode=0 Oct 30 20:06:08.620933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.620778 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerDied","Data":"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780"} Oct 30 20:06:08.621246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.621230 4959 scope.go:122] "RemoveContainer" containerID="dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780" Oct 30 20:06:08.623096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.623064 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:06:08.623174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.623145 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:06:08.623223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:08.623179 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:06:09.634547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:09.634480 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerStarted","Data":"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d"} Oct 30 20:06:09.675112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:09.675080 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:10.067581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:10.067337 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:06:10.074267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:10.074232 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:10.074453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:10.074288 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:10.994835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:10.994802 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.072755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:11.072714 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:11.072970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:11.072844 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:11.719415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:11.719384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.066579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:12.058710 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:12.066579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:12.058782 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:12.120137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:12.120107 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:06:13.072541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:13.051939 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.072541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:13.052009 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:13.401030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:13.401000 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.069954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:14.069913 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.071377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:14.071346 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:15.054815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:15.054766 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.055352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:15.054857 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 20:06:15.077028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:15.076995 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:16.004163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.004130 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:06:16.010518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.010473 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.017005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.016976 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:06:16.041661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.041622 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c17154\"/\"kube-root-ca.crt\"" Oct 30 20:06:16.041661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.041652 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c17154\"/\"test-model-c17154-kserve-self-signed-certs\"" Oct 30 20:06:16.041878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.041729 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c17154\"/\"openshift-service-ca.crt\"" Oct 30 20:06:16.041878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.041734 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c17154\"/\"default-dockercfg-p7954\"" Oct 30 20:06:16.045872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.045847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:16.055279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.055253 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:06:16.057478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.057454 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:06:16.071437 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:06:16.071406 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:06:16.110230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110196 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.110230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110236 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.110489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110256 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.110489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110372 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zncj7\" (UniqueName: \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.110620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110540 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.110721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.110706 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212440 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212481 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212520 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212561 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zncj7\" (UniqueName: \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212624 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.212798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212780 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.213033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212912 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.213033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.212953 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.213213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.213197 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.216083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.216058 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.216302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.216281 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.233723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.233682 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zncj7\" (UniqueName: \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\") pod \"test-model-c17154-kserve-7658cc6cb5-cp8j7\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.342685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.332781 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:16.724046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.724015 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:06:16.725738 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:06:16.725707 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb1a303_3f04_46c3_aa38_7ba67a642cfd.slice/crio-c2d8b8f7e8e87d956e7956b6202f5cd86ed34e9795b1c89418ad2049ff435ad6 WatchSource:0}: Error finding container c2d8b8f7e8e87d956e7956b6202f5cd86ed34e9795b1c89418ad2049ff435ad6: Status 404 returned error can't find the container with id c2d8b8f7e8e87d956e7956b6202f5cd86ed34e9795b1c89418ad2049ff435ad6 Oct 30 20:06:16.964042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:16.963996 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:17.730574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:17.730535 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" event={"ID":"feb1a303-3f04-46c3-aa38-7ba67a642cfd","Type":"ContainerStarted","Data":"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c"} Oct 30 20:06:17.731072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:17.730583 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" event={"ID":"feb1a303-3f04-46c3-aa38-7ba67a642cfd","Type":"ContainerStarted","Data":"c2d8b8f7e8e87d956e7956b6202f5cd86ed34e9795b1c89418ad2049ff435ad6"} Oct 30 20:06:17.778385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:17.778331 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" podStartSLOduration=2.778316364 podStartE2EDuration="2.778316364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:17.775983858 +0000 UTC m=+1901.905217780" watchObservedRunningTime="2026-10-30 20:06:17.778316364 +0000 UTC m=+1901.907550286" Oct 30 20:06:18.333199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:18.333167 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:18.701248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:18.701213 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:20.302502 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:20.302472 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:21.084520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:21.076981 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:22.284520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:22.284358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.283545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:24.283482 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.334329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:26.334297 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:26.340234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:26.340199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.347229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:26.347209 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:26.910028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:26.910000 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:06:28.084930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:28.084894 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:29.836561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:29.836525 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.368031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:30.368001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.826368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:30.826333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.950619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:30.950562 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.080952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:31.080875 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.280743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:31.280714 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.435250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:31.435219 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.244878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:32.244834 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.637702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:32.637659 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:33.205013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.204982 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:06:33.210034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.210008 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.231917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.231894 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:06:33.241413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.240789 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4a31d3\"/\"openshift-service-ca.crt\"" Oct 30 20:06:33.241413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.240802 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4a31d3\"/\"kube-root-ca.crt\"" Oct 30 20:06:33.260666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.260640 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4a31d3\"/\"test-model-4a31d3-kserve-self-signed-certs\"" Oct 30 20:06:33.261105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.260641 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4a31d3\"/\"default-dockercfg-5x49c\"" Oct 30 20:06:33.272890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.272866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:33.294033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294002 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.294192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294144 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.294241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294190 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.294293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294279 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.294473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294451 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.294624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.294609 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr6wp\" (UniqueName: \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396069 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396158 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396194 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396226 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396347 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396511 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr6wp\" (UniqueName: \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396668 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396735 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.396821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.396787 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.398672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.398647 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.398971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.398954 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.419461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.419437 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr6wp\" (UniqueName: \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\") pod \"test-model-4a31d3-kserve-c6959558c-c4d5s\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.528247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.528150 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:33.919381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:33.919348 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:06:33.919562 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:06:33.919468 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod488df13a_d848_449b_9d3a_6c275613ee4e.slice/crio-4b0e595c237ceb6679b85bdb09a7c579e8ea7c88d09751078f30a85fa3c57d26 WatchSource:0}: Error finding container 4b0e595c237ceb6679b85bdb09a7c579e8ea7c88d09751078f30a85fa3c57d26: Status 404 returned error can't find the container with id 4b0e595c237ceb6679b85bdb09a7c579e8ea7c88d09751078f30a85fa3c57d26 Oct 30 20:06:34.003190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:34.003156 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" event={"ID":"488df13a-d848-449b-9d3a-6c275613ee4e","Type":"ContainerStarted","Data":"4b0e595c237ceb6679b85bdb09a7c579e8ea7c88d09751078f30a85fa3c57d26"} Oct 30 20:06:34.624983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:34.624953 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:35.000932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:35.000898 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" event={"ID":"488df13a-d848-449b-9d3a-6c275613ee4e","Type":"ContainerStarted","Data":"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2"} Oct 30 20:06:35.064713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:35.064655 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" podStartSLOduration=2.064635361 podStartE2EDuration="2.064635361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:35.062070839 +0000 UTC m=+1919.191304759" watchObservedRunningTime="2026-10-30 20:06:35.064635361 +0000 UTC m=+1919.193869284" Oct 30 20:06:35.528328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:35.528299 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:35.928913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:35.928882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.062803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:36.062774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.711659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:36.711622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.679462 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:37.679425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.856653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:37.856622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:38.683715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:38.683684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.429163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:40.429134 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.593701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:40.593658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.900865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:40.900824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:41.774204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:41.774159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.830756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:42.830728 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:43.529141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:43.529105 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:43.542560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:43.542534 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:44.227647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:44.227617 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:06:44.984919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:44.984886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:45.195863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:45.195826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:45.315223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:45.315137 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:46.068429 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:06:46.068392 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:06:46.971981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:46.971951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.102130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.102097 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.382621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.382540 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:06:47.473965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.468897 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.492060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.491956 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a64be2\"" Oct 30 20:06:47.495912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.495880 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5vd\" (UniqueName: \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.496285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.496263 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.514844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.508180 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a64be2-dockercfg-9z7h2\"" Oct 30 20:06:47.535272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.535235 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:06:47.597712 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.597640 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5vd\" (UniqueName: \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.597905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.597841 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.598646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.598614 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.619423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.619392 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5vd\" (UniqueName: \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\") pod \"payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:47.834537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:47.834478 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:48.158486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:48.158461 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:06:48.159350 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:06:48.159323 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2306e0ae_3564_437e_a17d_b5e15ba074f6.slice/crio-b8672c6bf49da234259a830421109efc27ae21b09f6bb630d806dd4ccdf12bc3 WatchSource:0}: Error finding container b8672c6bf49da234259a830421109efc27ae21b09f6bb630d806dd4ccdf12bc3: Status 404 returned error can't find the container with id b8672c6bf49da234259a830421109efc27ae21b09f6bb630d806dd4ccdf12bc3 Oct 30 20:06:48.338531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:48.338468 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" event={"ID":"2306e0ae-3564-437e-a17d-b5e15ba074f6","Type":"ContainerStarted","Data":"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2"} Oct 30 20:06:48.338723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:48.338549 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" event={"ID":"2306e0ae-3564-437e-a17d-b5e15ba074f6","Type":"ContainerStarted","Data":"b8672c6bf49da234259a830421109efc27ae21b09f6bb630d806dd4ccdf12bc3"} Oct 30 20:06:48.338810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:48.338766 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:48.382291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:48.382237 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" podStartSLOduration=1.382219675 podStartE2EDuration="1.382219675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:48.381796625 +0000 UTC m=+1932.511030547" watchObservedRunningTime="2026-10-30 20:06:48.382219675 +0000 UTC m=+1932.511453596" Oct 30 20:06:49.129373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:49.129032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:50.803820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:50.803787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.594995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:52.594945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:54.290268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:54.290229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:55.992147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:55.992112 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:57.804197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:57.804169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.593824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:58.593790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.703169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:58.703130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:59.350144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:59.350109 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:06:59.381943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:06:59.381908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:01.085387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:01.085348 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:01.630867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:01.630823 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:07:01.702728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:01.702691 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.113570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:03.113532 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:05.699156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:05.699116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:07.626015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:07.625983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:09.546107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:09.546070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.310017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:11.309986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.480914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:11.480864 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:13.235869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:13.235817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:14.445533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:14.445461 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:07:14.466793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:14.466680 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:15.245975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:15.245945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:15.698767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:15.698732 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 20:07:15.699214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:15.698953 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" containerID="cri-o://95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3" gracePeriod=30 Oct 30 20:07:16.067439 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:16.067359 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:07:16.326472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.326390 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:07:16.331737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.331709 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.347621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.347594 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:07:16.351243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.351217 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6cd2ad-openshift-default-dockercfg-r568l\"" Oct 30 20:07:16.358867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.358843 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.361268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.361231 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:07:16.448986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.448958 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:07:16.484018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.483978 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484126 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.484174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484160 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484190 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484254 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnvr\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484376 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484480 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.484641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484579 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.484641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484621 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484697 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwnd\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.484808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484788 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484839 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.484931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484915 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.484992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.484977 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.485073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.485058 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.485179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.485159 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.485241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.485210 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.485378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.485353 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587274 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587312 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587334 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587351 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnvr\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587373 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587454 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587477 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587688 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587740 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwnd\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587769 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587778 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587795 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587831 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587835 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.587914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587853 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.587973 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588028 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588087 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588177 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588193 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.588281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588223 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.588575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588286 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588293 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588368 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.588744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588681 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.588821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.588793 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.589119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.589029 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.589236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.589211 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.590723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.590698 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.590723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.590714 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.590894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.590875 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.591056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.591033 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.615236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.615170 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwnd\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.615992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.615965 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.616141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.616073 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnvr\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.616374 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.616346 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.651123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.651064 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:16.670808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.670760 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" podUID="92972dd3-69d1-4316-afe5-6a58cd965d96" containerName="main" containerID="cri-o://ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c" gracePeriod=45 Oct 30 20:07:16.685065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.685039 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:16.973632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:16.973606 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:07:17.002652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.002625 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:07:17.004267 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:07:17.004245 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae28612_2090_4280_9139_63ef293ba821.slice/crio-7e3c85511d024d957b27f7b9a533e9f9c7b5ddc0a76e80eb87444ca97b911e8a WatchSource:0}: Error finding container 7e3c85511d024d957b27f7b9a533e9f9c7b5ddc0a76e80eb87444ca97b911e8a: Status 404 returned error can't find the container with id 7e3c85511d024d957b27f7b9a533e9f9c7b5ddc0a76e80eb87444ca97b911e8a Oct 30 20:07:17.007003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.006969 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.007074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.007059 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.007113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.007094 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.021298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.021274 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:07:17.025553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.025520 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.025656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.025593 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.025656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.025639 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:07:17.097644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.097618 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.097784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.097697 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.097853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.097784 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\" (UniqueName: \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.097853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.097816 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.097965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.097879 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\" (UniqueName: \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.098024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098004 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\") pod \"92972dd3-69d1-4316-afe5-6a58cd965d96\" (UID: \"92972dd3-69d1-4316-afe5-6a58cd965d96\") " Oct 30 20:07:17.098094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098036 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:17.098208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098190 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:17.098431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098407 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:17.098876 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098856 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.098952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098881 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.098952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.098890 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.101946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.101921 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "kube-api-access-nk6j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:17.102052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.102003 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:17.102052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.102034 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm" pod "92972dd3-69d1-4316-afe5-6a58cd965d96" (UID: "92972dd3-69d1-4316-afe5-6a58cd965d96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:17.200726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.200698 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6j8\" (UniqueName: \"kubernetes.io/projected/92972dd3-69d1-4316-afe5-6a58cd965d96-kube-api-access-nk6j8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.200726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.200725 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92972dd3-69d1-4316-afe5-6a58cd965d96-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.201005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.200737 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92972dd3-69d1-4316-afe5-6a58cd965d96-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:17.291274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.291144 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:17.693230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.693196 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" event={"ID":"440c558d-c52a-45f1-964e-bc090bd0268c","Type":"ContainerStarted","Data":"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f"} Oct 30 20:07:17.693403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.693257 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" event={"ID":"440c558d-c52a-45f1-964e-bc090bd0268c","Type":"ContainerStarted","Data":"deba3a06d4c5f61f243a7d3ffe37e8bffe380b111572c3290fb658c9040d8c21"} Oct 30 20:07:17.695638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.695610 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" event={"ID":"cae28612-2090-4280-9139-63ef293ba821","Type":"ContainerStarted","Data":"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc"} Oct 30 20:07:17.695809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.695660 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" event={"ID":"cae28612-2090-4280-9139-63ef293ba821","Type":"ContainerStarted","Data":"7e3c85511d024d957b27f7b9a533e9f9c7b5ddc0a76e80eb87444ca97b911e8a"} Oct 30 20:07:17.697192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.697168 4959 generic.go:356] "Generic (PLEG): container finished" podID="92972dd3-69d1-4316-afe5-6a58cd965d96" containerID="ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c" exitCode=0 Oct 30 20:07:17.697308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.697201 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" Oct 30 20:07:17.697308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.697215 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" event={"ID":"92972dd3-69d1-4316-afe5-6a58cd965d96","Type":"ContainerDied","Data":"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c"} Oct 30 20:07:17.697308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.697248 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg" event={"ID":"92972dd3-69d1-4316-afe5-6a58cd965d96","Type":"ContainerDied","Data":"16ca605bbed24c93f7dfe34d692b1d7899801729f0fdb39795a6659a29ec2ba0"} Oct 30 20:07:17.697308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.697271 4959 scope.go:122] "RemoveContainer" containerID="ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c" Oct 30 20:07:17.710966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.710939 4959 scope.go:122] "RemoveContainer" containerID="ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c" Oct 30 20:07:17.711352 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:17.711284 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c\": container with ID starting with ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c not found: ID does not exist" containerID="ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c" Oct 30 20:07:17.711352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.711312 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c"} err="failed to get container status \"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c\": rpc error: code = NotFound desc = could not find container \"ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c\": container with ID starting with ebcea60215ecc581f6d47da5a234384b69470bd4119f3c043ed0e18f7303f16c not found: ID does not exist" Oct 30 20:07:17.747115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.747077 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podStartSLOduration=1.747064811 podStartE2EDuration="1.747064811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:17.744262414 +0000 UTC m=+1961.873496336" watchObservedRunningTime="2026-10-30 20:07:17.747064811 +0000 UTC m=+1961.876298791" Oct 30 20:07:17.788421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.788385 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:07:17.792753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.792717 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0506c9/test-model-0506c9-kserve-7b6d585b9c-rgrpg"] Oct 30 20:07:17.838291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:17.838237 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podStartSLOduration=1.83822289 podStartE2EDuration="1.83822289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:17.837226282 +0000 UTC m=+1961.966460204" watchObservedRunningTime="2026-10-30 20:07:17.83822289 +0000 UTC m=+1961.967456810" Oct 30 20:07:18.001628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.001595 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92972dd3-69d1-4316-afe5-6a58cd965d96" path="/var/lib/kubelet/pods/92972dd3-69d1-4316-afe5-6a58cd965d96/volumes" Oct 30 20:07:18.702864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.684911 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:18.702864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.685305 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:18.702864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.697817 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.702864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.697881 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:18.704675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.704644 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.704834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:18.704694 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:19.119798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:19.119635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:19.669531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:19.668623 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:19.669531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:19.668688 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:19.692521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:19.686994 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:19.692521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:19.687057 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:20.669522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.667928 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.669522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.667995 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:20.692521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.687935 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.692521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.687991 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:20.840365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.840337 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:07:20.840595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.840390 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 20:07:20.975520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:20.972002 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:21.319220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.318924 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432131 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432235 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432316 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432349 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432400 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432422 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.432526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432440 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.433019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432613 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:21.433019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432657 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:21.433019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432732 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:21.433019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.432897 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.433172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.433084 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:21.433172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.433157 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\") pod \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\" (UID: \"2236e32f-fdda-42e5-84e7-a0b14a6f8f87\") " Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.433431 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434068 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434084 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434093 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434101 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.434783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434109 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.435759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.434896 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:21.435759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.435339 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "kube-api-access-4rhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:21.435879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.435834 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:07:21.436686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.436665 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token" pod "2236e32f-fdda-42e5-84e7-a0b14a6f8f87" (UID: "2236e32f-fdda-42e5-84e7-a0b14a6f8f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:21.537609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.537486 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.537609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.537559 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.537609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.537574 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rhm7\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-kube-api-access-4rhm7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.537609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.537591 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2236e32f-fdda-42e5-84e7-a0b14a6f8f87-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:21.675519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.674606 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:21.675519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.674664 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:21.698546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.698305 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:21.698546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.698363 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:21.741013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.740980 4959 generic.go:356] "Generic (PLEG): container finished" podID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerID="95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3" exitCode=0 Oct 30 20:07:21.741156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.741137 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" event={"ID":"2236e32f-fdda-42e5-84e7-a0b14a6f8f87","Type":"ContainerDied","Data":"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3"} Oct 30 20:07:21.741213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.741183 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" event={"ID":"2236e32f-fdda-42e5-84e7-a0b14a6f8f87","Type":"ContainerDied","Data":"47423c9986074a1be3c7ae95f3d0abedda43fd01011f9665af8c12d000bb2102"} Oct 30 20:07:21.741213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.741209 4959 scope.go:122] "RemoveContainer" containerID="95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3" Oct 30 20:07:21.741362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.741349 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf" Oct 30 20:07:21.800191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.800111 4959 scope.go:122] "RemoveContainer" containerID="95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3" Oct 30 20:07:21.800584 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:21.800554 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3\": container with ID starting with 95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3 not found: ID does not exist" containerID="95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3" Oct 30 20:07:21.800709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.800595 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3"} err="failed to get container status \"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3\": rpc error: code = NotFound desc = could not find container \"95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3\": container with ID starting with 95c8335ed16932df96de1dba4060324a6cc211f5357a82034b84cf3b5de82bd3 not found: ID does not exist" Oct 30 20:07:21.839888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.839854 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 20:07:21.844463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:21.844437 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-m2bbf"] Oct 30 20:07:22.011444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.011412 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" path="/var/lib/kubelet/pods/2236e32f-fdda-42e5-84e7-a0b14a6f8f87/volumes" Oct 30 20:07:22.667523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.663975 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:22.667523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.664059 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:22.690522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.689633 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:22.690522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.689697 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:22.731645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:22.731615 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:23.653523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:23.651910 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 20:07:23.653523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:23.651977 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 20:07:23.686135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:23.686087 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 30 20:07:23.686340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:23.686153 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 30 20:07:24.583957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.583914 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:24.702997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.702896 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:24.703950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.703927 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:24.704869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.704686 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:07:24.704869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.704769 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:24.715367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.705392 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:24.715367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:24.715010 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:07:25.171952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:25.171924 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 20:07:25.219682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:25.219655 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 20:07:25.357731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:25.357701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.488604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:25.483714 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.570182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.570141 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:07:26.571274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571250 4959 cpu_manager.go:414] "containerMap: removing container" podUID="92972dd3-69d1-4316-afe5-6a58cd965d96" containerName="main" Oct 30 20:07:26.571390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571275 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92972dd3-69d1-4316-afe5-6a58cd965d96" containerName="main" Oct 30 20:07:26.571390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571324 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" Oct 30 20:07:26.571390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571332 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" Oct 30 20:07:26.571616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571597 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="92972dd3-69d1-4316-afe5-6a58cd965d96" containerName="main" Oct 30 20:07:26.571672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.571622 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2236e32f-fdda-42e5-84e7-a0b14a6f8f87" containerName="istio-proxy" Oct 30 20:07:26.576739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.576718 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.597099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.597078 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:07:26.597743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.597723 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d3c508-serving-cert\"" Oct 30 20:07:26.650143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.650109 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.670017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.669986 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.670300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.670280 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.670352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.670312 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxmn\" (UniqueName: \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.670352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.670334 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.759350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.759317 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.771944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.771916 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.772063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.771945 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxmn\" (UniqueName: \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.772063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.771968 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.772063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.772034 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.772802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.772741 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.774596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.774566 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.774708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.774661 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.793409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.793386 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxmn\" (UniqueName: \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\") pod \"maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.902145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.899349 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:26.961405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:26.961363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:27.237067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:27.237035 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:07:27.237461 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:07:27.237436 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758dba28_c590_419b_b34c_402cad2c77e8.slice/crio-be832e9b96db64f84b993e4324e0ad695a6048767bf31d9bfffe8d001d12054a WatchSource:0}: Error finding container be832e9b96db64f84b993e4324e0ad695a6048767bf31d9bfffe8d001d12054a: Status 404 returned error can't find the container with id be832e9b96db64f84b993e4324e0ad695a6048767bf31d9bfffe8d001d12054a Oct 30 20:07:27.919900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:27.918233 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" event={"ID":"758dba28-c590-419b-b34c-402cad2c77e8","Type":"ContainerStarted","Data":"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca"} Oct 30 20:07:27.919900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:27.918300 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" event={"ID":"758dba28-c590-419b-b34c-402cad2c77e8","Type":"ContainerStarted","Data":"be832e9b96db64f84b993e4324e0ad695a6048767bf31d9bfffe8d001d12054a"} Oct 30 20:07:27.919900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:27.919613 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:27.990810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:27.990698 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" podStartSLOduration=1.5324464469999999 podStartE2EDuration="1.990683514s" totalImagesPullingTime="458.237067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:27.989154969 +0000 UTC m=+1972.118388891" watchObservedRunningTime="2026-10-30 20:07:27.990683514 +0000 UTC m=+1972.119917436" Oct 30 20:07:28.437461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:28.437423 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:28.556098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:28.556066 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:28.694381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:28.694297 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:29.469267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.469218 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" containerID="cri-o://4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1" gracePeriod=45 Oct 30 20:07:29.782255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.782230 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:07:29.864334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864289 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\" (UniqueName: \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.864555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864408 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.864689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864664 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.864830 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864716 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.864830 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864747 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.864830 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864764 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:29.864830 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864798 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\") pod \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\" (UID: \"6d04c0c3-68a1-4e3c-b04f-7795249608fd\") " Oct 30 20:07:29.865063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.864915 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:29.865708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.865420 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:29.865708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.865651 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:29.865708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.865671 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:29.865708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.865685 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:29.867691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.867660 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:29.867691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.867667 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "kube-api-access-cxn9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:29.867829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.867669 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs" pod "6d04c0c3-68a1-4e3c-b04f-7795249608fd" (UID: "6d04c0c3-68a1-4e3c-b04f-7795249608fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:29.935925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.935885 4959 generic.go:356] "Generic (PLEG): container finished" podID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerID="4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1" exitCode=0 Oct 30 20:07:29.936135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.935949 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" Oct 30 20:07:29.936135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.935974 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" event={"ID":"6d04c0c3-68a1-4e3c-b04f-7795249608fd","Type":"ContainerDied","Data":"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1"} Oct 30 20:07:29.936135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.936019 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps" event={"ID":"6d04c0c3-68a1-4e3c-b04f-7795249608fd","Type":"ContainerDied","Data":"4e1847c93732d4edd95c7c4e49cd2dea8125529fc9aa86162dc4d3309d679023"} Oct 30 20:07:29.936135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.936036 4959 scope.go:122] "RemoveContainer" containerID="4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1" Oct 30 20:07:29.949042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.949022 4959 scope.go:122] "RemoveContainer" containerID="4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1" Oct 30 20:07:29.949361 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:29.949332 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1\": container with ID starting with 4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1 not found: ID does not exist" containerID="4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1" Oct 30 20:07:29.949420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.949360 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1"} err="failed to get container status \"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1\": rpc error: code = NotFound desc = could not find container \"4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1\": container with ID starting with 4d104f363c06d4cca6a115799b117e02cdc8c7777e1cd7458768fe6b30ae8db1 not found: ID does not exist" Oct 30 20:07:29.967553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.967519 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxn9v\" (UniqueName: \"kubernetes.io/projected/6d04c0c3-68a1-4e3c-b04f-7795249608fd-kube-api-access-cxn9v\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:29.967717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.967651 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d04c0c3-68a1-4e3c-b04f-7795249608fd-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:29.967717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:29.967671 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d04c0c3-68a1-4e3c-b04f-7795249608fd-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:30.009001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.008921 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:07:30.009001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.008965 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eb16ce/test-model-eb16ce-kserve-5b486c69c9-5hcps"] Oct 30 20:07:30.172201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.172155 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" containerID="cri-o://09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f" gracePeriod=25 Oct 30 20:07:30.220289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.220248 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" containerID="cri-o://e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d" gracePeriod=25 Oct 30 20:07:30.383069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.383028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:30.938804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:30.938764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:31.418705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:31.418673 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:32.001897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:32.001861 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" path="/var/lib/kubelet/pods/6d04c0c3-68a1-4e3c-b04f-7795249608fd/volumes" Oct 30 20:07:32.105459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:32.105430 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:33.821156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:33.821122 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:34.942320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:34.942290 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:07:35.465465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:35.465431 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.078932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.078904 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:07:37.079841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.079824 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" Oct 30 20:07:37.079892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.079842 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" Oct 30 20:07:37.080017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.080007 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d04c0c3-68a1-4e3c-b04f-7795249608fd" containerName="main" Oct 30 20:07:37.085581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.085556 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.105243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.105199 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:07:37.111965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.111940 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a64be2-serving-cert\"" Oct 30 20:07:37.173615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.173530 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.211098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.211065 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.211293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.211134 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5p9\" (UniqueName: \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.211365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.211349 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.211418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.211395 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.317057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.318617 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.318732 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5p9\" (UniqueName: \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.318987 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.319056 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:37.319232 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-serving-cert: secret "maas-api-e2e-inf-a-a64be2-serving-cert" not found Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:37.319297 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls podName:7bf8063e-366b-4bf8-926f-22d93e41be8a nodeName:}" failed. No retries permitted until 2026-10-30 20:07:37.819275331 +0000 UTC m=+1981.948509247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls") pod "maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" (UID: "7bf8063e-366b-4bf8-926f-22d93e41be8a") : secret "maas-api-e2e-inf-a-a64be2-serving-cert" not found Oct 30 20:07:37.328519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.320277 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.371032 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.370982 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5p9\" (UniqueName: \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.371816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.371734 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.488997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.488906 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.500965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.500928 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:37.500965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.500956 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:37.827676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.827586 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:37.830408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:37.830379 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") pod \"maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:38.044522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:38.028109 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:38.502904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:38.502877 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:07:39.046942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:39.046814 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" event={"ID":"7bf8063e-366b-4bf8-926f-22d93e41be8a","Type":"ContainerStarted","Data":"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987"} Oct 30 20:07:39.046942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:39.046890 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" event={"ID":"7bf8063e-366b-4bf8-926f-22d93e41be8a","Type":"ContainerStarted","Data":"b0311d96a6243c81d1d3be8c256d8ce97d1ea59cee4696198dad40abaacdcd15"} Oct 30 20:07:39.046942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:39.046919 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:39.092406 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:39.092346 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" podStartSLOduration=1.830937675 podStartE2EDuration="2.092331521s" totalImagesPullingTime="261.393846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:39.091038009 +0000 UTC m=+1983.220271942" watchObservedRunningTime="2026-10-30 20:07:39.092331521 +0000 UTC m=+1983.221565443" Oct 30 20:07:39.178726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:39.178688 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.867923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:40.867886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.492397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:42.492364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:44.289218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:44.289184 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:45.062287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:45.062257 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:07:45.261384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:45.261353 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 20:07:45.261570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:45.261548 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" containerID="cri-o://08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13" gracePeriod=30 Oct 30 20:07:45.996629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:45.994183 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:46.152204 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:46.152168 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:07:47.500163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:47.500127 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:47.500620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:47.500126 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:07:47.726081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:47.726037 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:49.313447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:49.313414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:50.342805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.342760 4959 patch_prober.go:29] interesting pod/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:07:50.343139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.342855 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 20:07:50.545221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.545187 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 20:07:50.576540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.576479 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 20:07:50.626807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.626781 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 20:07:50.713238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713204 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\") pod \"ef10935a-addd-444a-b27e-ae3e43183cb2\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " Oct 30 20:07:50.713422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713254 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\" (UniqueName: \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\") pod \"392ba427-9917-49bf-8d60-8459a583f4e8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " Oct 30 20:07:50.713422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713359 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\" (UniqueName: \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\") pod \"ef10935a-addd-444a-b27e-ae3e43183cb2\" (UID: \"ef10935a-addd-444a-b27e-ae3e43183cb2\") " Oct 30 20:07:50.713572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713556 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\") pod \"392ba427-9917-49bf-8d60-8459a583f4e8\" (UID: \"392ba427-9917-49bf-8d60-8459a583f4e8\") " Oct 30 20:07:50.713705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713678 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume" pod "ef10935a-addd-444a-b27e-ae3e43183cb2" (UID: "ef10935a-addd-444a-b27e-ae3e43183cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:50.713978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.713952 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume" pod "392ba427-9917-49bf-8d60-8459a583f4e8" (UID: "392ba427-9917-49bf-8d60-8459a583f4e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:50.714436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.714412 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef10935a-addd-444a-b27e-ae3e43183cb2-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.714519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.714444 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392ba427-9917-49bf-8d60-8459a583f4e8-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.716357 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.716334 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd" pod "ef10935a-addd-444a-b27e-ae3e43183cb2" (UID: "ef10935a-addd-444a-b27e-ae3e43183cb2"). InnerVolumeSpecName "kube-api-access-qdltd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:50.716447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.716367 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g" pod "392ba427-9917-49bf-8d60-8459a583f4e8" (UID: "392ba427-9917-49bf-8d60-8459a583f4e8"). InnerVolumeSpecName "kube-api-access-zr52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:50.815254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815133 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815265 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815300 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815333 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815414 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:50.815695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815426 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815524 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815586 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815646 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815772 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\") pod \"de39728e-03b2-43af-9df8-3106cb64fa61\" (UID: \"de39728e-03b2-43af-9df8-3106cb64fa61\") " Oct 30 20:07:50.815903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815809 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:50.815903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.815829 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:50.816134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.816104 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:50.816372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.816336 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:50.817062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817040 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817071 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de39728e-03b2-43af-9df8-3106cb64fa61-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817086 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817099 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817114 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817126 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr52g\" (UniqueName: \"kubernetes.io/projected/392ba427-9917-49bf-8d60-8459a583f4e8-kube-api-access-zr52g\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817139 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdltd\" (UniqueName: \"kubernetes.io/projected/ef10935a-addd-444a-b27e-ae3e43183cb2-kube-api-access-qdltd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.817976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.817946 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:50.818309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.818288 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:07:50.818619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.818602 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:07:50.818691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.818631 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc" pod "de39728e-03b2-43af-9df8-3106cb64fa61" (UID: "de39728e-03b2-43af-9df8-3106cb64fa61"). InnerVolumeSpecName "kube-api-access-mqltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:50.917977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.917953 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de39728e-03b2-43af-9df8-3106cb64fa61-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.917977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.917977 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqltc\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-kube-api-access-mqltc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.918146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.917987 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de39728e-03b2-43af-9df8-3106cb64fa61-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:50.918146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:50.917996 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de39728e-03b2-43af-9df8-3106cb64fa61-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:51.089400 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.089311 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:51.196036 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.196003 4959 generic.go:356] "Generic (PLEG): container finished" podID="de39728e-03b2-43af-9df8-3106cb64fa61" containerID="08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13" exitCode=0 Oct 30 20:07:51.196256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.196138 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" event={"ID":"de39728e-03b2-43af-9df8-3106cb64fa61","Type":"ContainerDied","Data":"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13"} Oct 30 20:07:51.196256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.196193 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" event={"ID":"de39728e-03b2-43af-9df8-3106cb64fa61","Type":"ContainerDied","Data":"f93f27209ceee32bf8ab5bfaa3b7915536adbf0a818cfa43ee3727e4566c139b"} Oct 30 20:07:51.196256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.196224 4959 scope.go:122] "RemoveContainer" containerID="08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13" Oct 30 20:07:51.196582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.196392 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn" Oct 30 20:07:51.200734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.200705 4959 generic.go:356] "Generic (PLEG): container finished" podID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerID="09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f" exitCode=0 Oct 30 20:07:51.201201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.200918 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" event={"ID":"ef10935a-addd-444a-b27e-ae3e43183cb2","Type":"ContainerDied","Data":"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f"} Oct 30 20:07:51.201201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.200973 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" event={"ID":"ef10935a-addd-444a-b27e-ae3e43183cb2","Type":"ContainerDied","Data":"2f3080f07360a3b0f19fa3628111737fd70e0307a8de1b47db0d8c8de5f52e74"} Oct 30 20:07:51.201201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.201091 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6" Oct 30 20:07:51.206769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.205012 4959 generic.go:356] "Generic (PLEG): container finished" podID="392ba427-9917-49bf-8d60-8459a583f4e8" containerID="e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d" exitCode=0 Oct 30 20:07:51.206769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.205140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" event={"ID":"392ba427-9917-49bf-8d60-8459a583f4e8","Type":"ContainerDied","Data":"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d"} Oct 30 20:07:51.206769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.205174 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" event={"ID":"392ba427-9917-49bf-8d60-8459a583f4e8","Type":"ContainerDied","Data":"e4778243a7f8bd8bd6338477304b3a40a07838b8485af459d1f6aebde7b58034"} Oct 30 20:07:51.206769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.205206 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8" Oct 30 20:07:51.236375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.236179 4959 scope.go:122] "RemoveContainer" containerID="08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13" Oct 30 20:07:51.236595 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:51.236568 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13\": container with ID starting with 08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13 not found: ID does not exist" containerID="08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13" Oct 30 20:07:51.236818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.236606 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13"} err="failed to get container status \"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13\": rpc error: code = NotFound desc = could not find container \"08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13\": container with ID starting with 08b22d03d8fc81a2f172421e80ad29a6ebba4958100f0ffeb82f66c7a48dac13 not found: ID does not exist" Oct 30 20:07:51.236818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.236628 4959 scope.go:122] "RemoveContainer" containerID="09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f" Oct 30 20:07:51.247460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.247440 4959 scope.go:122] "RemoveContainer" containerID="09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f" Oct 30 20:07:51.247787 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:51.247763 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f\": container with ID starting with 09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f not found: ID does not exist" containerID="09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f" Oct 30 20:07:51.247863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.247797 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f"} err="failed to get container status \"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f\": rpc error: code = NotFound desc = could not find container \"09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f\": container with ID starting with 09ef04069ca946b96c5e333e0a93f85f25203d2629d030b219f2583a45396f9f not found: ID does not exist" Oct 30 20:07:51.247863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.247814 4959 scope.go:122] "RemoveContainer" containerID="e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d" Oct 30 20:07:51.258063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.258045 4959 scope.go:122] "RemoveContainer" containerID="e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d" Oct 30 20:07:51.258326 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:51.258308 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d\": container with ID starting with e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d not found: ID does not exist" containerID="e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d" Oct 30 20:07:51.258391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.258332 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d"} err="failed to get container status \"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d\": rpc error: code = NotFound desc = could not find container \"e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d\": container with ID starting with e38171d331fd9258608a855a6202ce0416df5589425885b2ae3c4579e3a9093d not found: ID does not exist" Oct 30 20:07:51.298291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.298264 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 20:07:51.302455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.302436 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0506c9-85d68d484c-4jln8"] Oct 30 20:07:51.344314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.344246 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 20:07:51.357811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.357786 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0506c9-7fc99bf86-wqjf6"] Oct 30 20:07:51.382225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.382200 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 20:07:51.386094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:51.386063 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-8cjsn"] Oct 30 20:07:52.005813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:52.005777 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" path="/var/lib/kubelet/pods/392ba427-9917-49bf-8d60-8459a583f4e8/volumes" Oct 30 20:07:52.006541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:52.006520 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" path="/var/lib/kubelet/pods/de39728e-03b2-43af-9df8-3106cb64fa61/volumes" Oct 30 20:07:52.007127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:52.007108 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" path="/var/lib/kubelet/pods/ef10935a-addd-444a-b27e-ae3e43183cb2/volumes" Oct 30 20:07:52.746472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:52.746432 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:54.388875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:54.388825 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:56.199375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:56.199328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:57.606651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:57.606623 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 20:07:57.607106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:57.606815 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" podUID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" containerName="maas-api" containerID="cri-o://034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e" gracePeriod=30 Oct 30 20:07:57.670428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:57.670374 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:58.230349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.230317 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 20:07:58.317542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.315898 4959 generic.go:356] "Generic (PLEG): container finished" podID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" containerID="034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e" exitCode=0 Oct 30 20:07:58.317542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.315954 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" Oct 30 20:07:58.317542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.316033 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" event={"ID":"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca","Type":"ContainerDied","Data":"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e"} Oct 30 20:07:58.317542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.316096 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq" event={"ID":"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca","Type":"ContainerDied","Data":"c0c4db68a8670c5b095499a827f8e2d773dd4ca0f1e771d5565753a0e8c4f7a5"} Oct 30 20:07:58.317542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.316132 4959 scope.go:122] "RemoveContainer" containerID="034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e" Oct 30 20:07:58.330874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.330856 4959 scope.go:122] "RemoveContainer" containerID="034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e" Oct 30 20:07:58.333836 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:07:58.333760 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e\": container with ID starting with 034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e not found: ID does not exist" containerID="034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e" Oct 30 20:07:58.333836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.333795 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e"} err="failed to get container status \"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e\": rpc error: code = NotFound desc = could not find container \"034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e\": container with ID starting with 034b083a0b22cedc9c60ab80c3f6a8646780fa30a349cfa54db8c0f454342c2e not found: ID does not exist" Oct 30 20:07:58.369722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.369634 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\" (UniqueName: \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\") pod \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " Oct 30 20:07:58.369888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.369786 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\") pod \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " Oct 30 20:07:58.369888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.369830 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\") pod \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " Oct 30 20:07:58.370043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.369945 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") pod \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\" (UID: \"2ca34760-24bb-491a-a5d6-e6ae80e7c5ca\") " Oct 30 20:07:58.370191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.370165 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca" pod "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" (UID: "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:58.370636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.370611 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.371969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.371939 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg" pod "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" (UID: "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca"). InnerVolumeSpecName "kube-api-access-584vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:58.372373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.372349 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs" pod "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" (UID: "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:58.372585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.372565 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls" pod "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" (UID: "2ca34760-24bb-491a-a5d6-e6ae80e7c5ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:58.472226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.472192 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.472226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.472226 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.472444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.472239 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584vg\" (UniqueName: \"kubernetes.io/projected/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca-kube-api-access-584vg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:07:58.684698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.684666 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 20:07:58.697652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.697620 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0506c9-5759855d8f-85tpq"] Oct 30 20:07:58.746873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.746812 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:58.973449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:58.973368 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:59.198970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:59.198936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:59.584576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:07:59.584543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:00.002985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:00.002951 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" path="/var/lib/kubelet/pods/2ca34760-24bb-491a-a5d6-e6ae80e7c5ca/volumes" Oct 30 20:08:01.011605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.011570 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:01.447777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.447732 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:08:01.448628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448609 4959 cpu_manager.go:414] "containerMap: removing container" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" Oct 30 20:08:01.448628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448626 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448674 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" containerName="maas-api" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448680 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" containerName="maas-api" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448699 4959 cpu_manager.go:414] "containerMap: removing container" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448704 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448729 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" Oct 30 20:08:01.448796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448734 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" Oct 30 20:08:01.448982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448954 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca34760-24bb-491a-a5d6-e6ae80e7c5ca" containerName="maas-api" Oct 30 20:08:01.448982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448977 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="de39728e-03b2-43af-9df8-3106cb64fa61" containerName="istio-proxy" Oct 30 20:08:01.449082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.448995 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef10935a-addd-444a-b27e-ae3e43183cb2" containerName="payload-processing" Oct 30 20:08:01.449082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.449012 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="392ba427-9917-49bf-8d60-8459a583f4e8" containerName="payload-pre-processing" Oct 30 20:08:01.454466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.454440 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.477641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.477609 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:08:01.521741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.521708 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.521936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.521876 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522026 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfhn\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522070 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522161 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522254 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522310 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522354 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.522423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.522407 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624538 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624663 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624733 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfhn\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624763 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624821 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624881 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.624957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624926 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624946 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.624970 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.625013 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.625152 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.625235 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.625365 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.625452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.625396 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.627666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.627641 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.627795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.627752 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.649798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.649774 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.650037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.650015 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfhn\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\") pod \"e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:01.776546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:01.776418 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:02.107873 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:08:02.107844 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c3ac53_79b3_4839_8679_7bea0f99d22a.slice/crio-f70011e73b9aace9a62ed251a7ba49bdd49428b3224eccfe8fec80833d53f58d WatchSource:0}: Error finding container f70011e73b9aace9a62ed251a7ba49bdd49428b3224eccfe8fec80833d53f58d: Status 404 returned error can't find the container with id f70011e73b9aace9a62ed251a7ba49bdd49428b3224eccfe8fec80833d53f58d Oct 30 20:08:02.108200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.108168 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:08:02.110717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.110683 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:02.110806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.110761 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:02.110806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.110793 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:02.368459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.368369 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" event={"ID":"41c3ac53-79b3-4839-8679-7bea0f99d22a","Type":"ContainerStarted","Data":"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df"} Oct 30 20:08:02.368459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.368450 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" event={"ID":"41c3ac53-79b3-4839-8679-7bea0f99d22a","Type":"ContainerStarted","Data":"f70011e73b9aace9a62ed251a7ba49bdd49428b3224eccfe8fec80833d53f58d"} Oct 30 20:08:02.429308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.429243 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podStartSLOduration=1.429225443 podStartE2EDuration="1.429225443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:02.427474823 +0000 UTC m=+2006.556708744" watchObservedRunningTime="2026-10-30 20:08:02.429225443 +0000 UTC m=+2006.558459365" Oct 30 20:08:02.747063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.747034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:02.782523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:02.777217 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:03.778955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:03.778922 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:03.779390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:03.778985 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:04.625779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:04.625748 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:04.781557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:04.778564 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:04.781557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:04.778640 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:05.777147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:05.777113 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:05.777367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:05.777175 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:06.772522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:06.771414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:06.777095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:06.777064 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:06.777233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:06.777133 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:06.943062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:06.942056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:07.189084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:07.189050 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:07.777656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:07.777621 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:07.778182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:07.777684 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:08.777143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:08.777106 4959 patch_prober.go:29] interesting pod/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 20:08:08.777352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:08.777180 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 20:08:09.559448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:09.559414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:09.781918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:09.781880 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:09.782289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:09.782266 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:09.783131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:09.783110 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:08:11.646456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:11.646422 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:13.528351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:13.528309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:16.168522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:16.162917 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:16.226521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:16.225595 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:16.281041 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:08:16.280775 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:08:16.522518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:16.522403 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:16.825578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:16.819583 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:16.901613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:16.901570 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:18.178098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:18.178065 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:18.296084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:18.296040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:18.825246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:18.825196 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:21.032057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:21.032021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:23.149468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:23.149418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:23.580517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:23.580420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:23.848819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:23.848737 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:24.127755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:24.127669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:25.276631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:25.276570 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:26.566747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.566707 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:08:26.573331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.573299 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.608414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.608371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:08:26.618626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.618594 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8fc40\"/\"test-model-c8fc40-kserve-self-signed-certs\"" Oct 30 20:08:26.618793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.618668 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8fc40\"/\"default-dockercfg-8xz26\"" Oct 30 20:08:26.618793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.618741 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8fc40\"/\"openshift-service-ca.crt\"" Oct 30 20:08:26.618793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.618756 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8fc40\"/\"kube-root-ca.crt\"" Oct 30 20:08:26.643564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.643524 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:26.716298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716264 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.716477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716319 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.716477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716393 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xvm\" (UniqueName: \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.716595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716577 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.716702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716685 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.716751 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.716737 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818185 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818230 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xvm\" (UniqueName: \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818381 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818593 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818604 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818780 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.818966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.818877 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.819387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.819366 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.821218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.821195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.821309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.821285 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.844142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.844113 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xvm\" (UniqueName: \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\") pod \"test-model-c8fc40-kserve-54fb88954c-726w4\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:26.893065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:26.893028 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:27.342894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.342859 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:08:27.756432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.756397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:27.768232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.768197 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" event={"ID":"7dc49ab8-dc5a-45ec-876e-461391f06947","Type":"ContainerStarted","Data":"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545"} Oct 30 20:08:27.768385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.768242 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" event={"ID":"7dc49ab8-dc5a-45ec-876e-461391f06947","Type":"ContainerStarted","Data":"7f9e5839590f685b3c14c1f6bd0af7880c594b44b55e4ae113f694b75ea94ee8"} Oct 30 20:08:27.859526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.859446 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" podStartSLOduration=1.8594218439999999 podStartE2EDuration="1.859421844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:27.822914673 +0000 UTC m=+2031.952148595" watchObservedRunningTime="2026-10-30 20:08:27.859421844 +0000 UTC m=+2031.988655771" Oct 30 20:08:27.893711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:27.893682 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:29.717302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:29.717267 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:31.058760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:31.058689 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:31.382640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:31.382486 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:31.525960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:31.525923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:33.135062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:33.135029 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:35.474258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:35.474223 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:36.894435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:36.894405 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:36.907971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:36.907941 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:37.449181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:37.449146 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:37.896149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:37.896123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:37.900785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:37.900766 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:08:38.370381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.370299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:38.842333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.842296 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:08:38.849052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.849025 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:38.873664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.873632 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:08:38.880538 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.879829 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4cfcee-openshift-default-dockercfg-tbq28\"" Oct 30 20:08:38.895044 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.894897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995331 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995412 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995472 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995726 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995799 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995859 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.995936 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmsf\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.996014 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.003522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:38.996044 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.113572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.097598 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.114201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.114166 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.147893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.142802 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.148342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.148313 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.148732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.148712 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmsf\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.149052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.149035 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.149194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.149180 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.149305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.149292 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.149522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.149439 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.150207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.115816 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.150368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.150351 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.151199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.151173 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.151316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.114905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.152143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.152110 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.161779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.161750 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.174174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.168899 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.201550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.193275 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmsf\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.232378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.231543 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\") pod \"e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.470159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.470107 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:39.839584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.839218 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:08:39.841450 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.840366 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:39.842659 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:08:39.842632 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6a0f85_0b17_4393_a20c_00da2257498a.slice/crio-175fb88687b369862bcef2f33450c976efb3de9c8537cedee9b691218fb5cc86 WatchSource:0}: Error finding container 175fb88687b369862bcef2f33450c976efb3de9c8537cedee9b691218fb5cc86: Status 404 returned error can't find the container with id 175fb88687b369862bcef2f33450c976efb3de9c8537cedee9b691218fb5cc86 Oct 30 20:08:39.846986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.846949 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:39.847257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.847024 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:39.847257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.847057 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:08:39.921401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:39.921369 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerStarted","Data":"175fb88687b369862bcef2f33450c976efb3de9c8537cedee9b691218fb5cc86"} Oct 30 20:08:40.290301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:40.289061 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:40.931673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:40.931639 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerStarted","Data":"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1"} Oct 30 20:08:41.007078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:41.007016 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podStartSLOduration=3.007003317 podStartE2EDuration="3.007003317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:41.003116672 +0000 UTC m=+2045.132350594" watchObservedRunningTime="2026-10-30 20:08:41.007003317 +0000 UTC m=+2045.136237237" Oct 30 20:08:41.470637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:41.470603 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:41.476968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:41.476938 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:41.941337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:41.941302 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:41.942397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:41.942374 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:08:42.369592 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:42.365057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:43.482062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:43.481996 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:43.612366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:43.612334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:44.464689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:44.464656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.333793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.333734 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.447237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.447197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.465981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.465949 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:08:45.470948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.470930 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.489784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.489751 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:08:45.498311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.498280 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4cfcee\"" Oct 30 20:08:45.498471 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.498408 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4cfcee-dockercfg-45wgz\"" Oct 30 20:08:45.615578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.587721 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl8t\" (UniqueName: \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.711960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.711922 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.712385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.703829 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:08:45.755439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.754196 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:08:45.755439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.754325 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.827509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.827440 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8db8t\" (UniqueName: \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.834156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.834124 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.835403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.835379 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl8t\" (UniqueName: \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.837642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.837617 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.838024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.835175 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.862294 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.862258 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl8t\" (UniqueName: \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\") pod \"payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:45.939690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.939657 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.939908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.939838 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8db8t\" (UniqueName: \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.940440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.940421 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:45.964692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:45.964657 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8db8t\" (UniqueName: \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\") pod \"payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:46.073323 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:08:46.073247 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:08:46.087416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.087380 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:46.141031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.140997 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:46.404886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.404854 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:08:46.405293 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:08:46.405269 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b2f127_2014_402e_8c65_bf0e85972e88.slice/crio-1560c9e6c571c1a7c236463ff81394bf3492f4159735a0cb42cbb3bf0584d6bd WatchSource:0}: Error finding container 1560c9e6c571c1a7c236463ff81394bf3492f4159735a0cb42cbb3bf0584d6bd: Status 404 returned error can't find the container with id 1560c9e6c571c1a7c236463ff81394bf3492f4159735a0cb42cbb3bf0584d6bd Oct 30 20:08:46.476707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.476681 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:08:46.485200 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:08:46.485172 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0462ecf_61b4_4a94_9fa9_72b9ac5403da.slice/crio-ad630a4b23ad0a727bd96ff0eceb2ae42268121c43a97b95b2f9fb25c0401ebe WatchSource:0}: Error finding container ad630a4b23ad0a727bd96ff0eceb2ae42268121c43a97b95b2f9fb25c0401ebe: Status 404 returned error can't find the container with id ad630a4b23ad0a727bd96ff0eceb2ae42268121c43a97b95b2f9fb25c0401ebe Oct 30 20:08:46.805232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.805150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:46.984174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.984132 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" event={"ID":"a0462ecf-61b4-4a94-9fa9-72b9ac5403da","Type":"ContainerStarted","Data":"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88"} Oct 30 20:08:46.984363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.984185 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" event={"ID":"a0462ecf-61b4-4a94-9fa9-72b9ac5403da","Type":"ContainerStarted","Data":"ad630a4b23ad0a727bd96ff0eceb2ae42268121c43a97b95b2f9fb25c0401ebe"} Oct 30 20:08:46.984363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:46.984216 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:47.000555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:47.000523 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" event={"ID":"13b2f127-2014-402e-8c65-bf0e85972e88","Type":"ContainerStarted","Data":"d5aec4d0d01cbc327881e9f10a518d43c9a6c12993be1c503cbc8549bfeb1790"} Oct 30 20:08:47.000795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:47.000575 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" event={"ID":"13b2f127-2014-402e-8c65-bf0e85972e88","Type":"ContainerStarted","Data":"1560c9e6c571c1a7c236463ff81394bf3492f4159735a0cb42cbb3bf0584d6bd"} Oct 30 20:08:47.000795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:47.000632 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:47.030723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:47.030665 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" podStartSLOduration=2.030647528 podStartE2EDuration="2.030647528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:47.02757422 +0000 UTC m=+2051.156808139" watchObservedRunningTime="2026-10-30 20:08:47.030647528 +0000 UTC m=+2051.159881446" Oct 30 20:08:47.075103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:47.075001 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" podStartSLOduration=2.074983756 podStartE2EDuration="2.074983756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:47.071475001 +0000 UTC m=+2051.200708922" watchObservedRunningTime="2026-10-30 20:08:47.074983756 +0000 UTC m=+2051.204217680" Oct 30 20:08:48.488022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:48.487972 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:50.284336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:50.284263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:52.155295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:52.155263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:53.999848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:53.999782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:55.807059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:55.807011 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:57.631404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:57.631370 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:58.011171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:58.011143 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:08:58.011561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:58.011542 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:08:59.413267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:08:59.413235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:00.797373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:00.797341 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 20:09:00.797881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:00.797553 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" containerID="cri-o://932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc" gracePeriod=30 Oct 30 20:09:01.337459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:01.337410 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.266756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.266724 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.609725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.609644 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:09:03.616540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.616487 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.637222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.637194 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:09:03.648526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.643008 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d3c508\"/\"openshift-service-ca.crt\"" Oct 30 20:09:03.648526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.643044 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d3c508\"/\"kube-root-ca.crt\"" Oct 30 20:09:03.648526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.643115 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d3c508\"/\"test-model-d3c508-kserve-self-signed-certs\"" Oct 30 20:09:03.649191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.649162 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d3c508\"/\"default-dockercfg-vttxs\"" Oct 30 20:09:03.674277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.674232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.684223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684180 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2vt\" (UniqueName: \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.684464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684252 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.684640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684612 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.684718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684696 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.684783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684736 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.684783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.684776 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786481 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2vt\" (UniqueName: \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786588 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786686 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786731 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786755 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.786910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.786783 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.787164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.787142 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.787164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.787154 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.787293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.787199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.789273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.789249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.789437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.789421 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.806572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.806540 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2vt\" (UniqueName: \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\") pod \"test-model-d3c508-kserve-58f9c95bc7-58t9m\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:03.936373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:03.936252 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:04.333312 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:09:04.333277 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f5d34fc_9370_46ea_98ec_d3ee86d8cbf6.slice/crio-de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518 WatchSource:0}: Error finding container de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518: Status 404 returned error can't find the container with id de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518 Oct 30 20:09:04.341231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:04.341209 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:09:05.197109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:05.197067 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" event={"ID":"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6","Type":"ContainerStarted","Data":"0617611d85df381290f39456b407683d0662d42d4d66a0d4abc9170461312cc2"} Oct 30 20:09:05.197284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:05.197121 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" event={"ID":"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6","Type":"ContainerStarted","Data":"de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518"} Oct 30 20:09:05.245183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:05.245135 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" podStartSLOduration=2.24512188 podStartE2EDuration="2.24512188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:05.240786715 +0000 UTC m=+2069.370020659" watchObservedRunningTime="2026-10-30 20:09:05.24512188 +0000 UTC m=+2069.374355801" Oct 30 20:09:05.540227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:05.540146 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:05.937601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:05.937556 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:06.121245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.121219 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 20:09:06.162951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.162922 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163020 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163079 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163123 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163190 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.163234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163214 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163330 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163354 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.163411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163395 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163521 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:06.163658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163578 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163604 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\") pod \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\" (UID: \"cca576ba-0a1a-457c-be3a-c3bbaac08f0c\") " Oct 30 20:09:06.163754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163673 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.163948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.163927 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.164422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.164404 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.164547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.164426 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.164547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.164440 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.164547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.164452 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.164547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.164468 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.165985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.165962 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:06.166088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.166069 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.166174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.166153 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "kube-api-access-b5ddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:06.166227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.166175 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo" pod "cca576ba-0a1a-457c-be3a-c3bbaac08f0c" (UID: "cca576ba-0a1a-457c-be3a-c3bbaac08f0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:06.208441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.208403 4959 generic.go:356] "Generic (PLEG): container finished" podID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerID="932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc" exitCode=0 Oct 30 20:09:06.208699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.208468 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" Oct 30 20:09:06.208699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.208583 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" event={"ID":"cca576ba-0a1a-457c-be3a-c3bbaac08f0c","Type":"ContainerDied","Data":"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc"} Oct 30 20:09:06.208699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.208663 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd" event={"ID":"cca576ba-0a1a-457c-be3a-c3bbaac08f0c","Type":"ContainerDied","Data":"8fab0cdea99a1a302790e1a10a778eb3265f14a89f62c486c47d13bb0503e064"} Oct 30 20:09:06.208899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.208709 4959 scope.go:122] "RemoveContainer" containerID="932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc" Oct 30 20:09:06.225001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.224965 4959 scope.go:122] "RemoveContainer" containerID="932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc" Oct 30 20:09:06.225694 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:09:06.225397 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc\": container with ID starting with 932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc not found: ID does not exist" containerID="932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc" Oct 30 20:09:06.225694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.225436 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc"} err="failed to get container status \"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc\": rpc error: code = NotFound desc = could not find container \"932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc\": container with ID starting with 932eed2ded4f3716fe24ac255ace45b2c9fc4d819f93615ef59290738aa826cc not found: ID does not exist" Oct 30 20:09:06.267252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.267200 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.267252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.267237 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.267252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.267256 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.267555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.267269 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5ddw\" (UniqueName: \"kubernetes.io/projected/cca576ba-0a1a-457c-be3a-c3bbaac08f0c-kube-api-access-b5ddw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:06.281413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.281383 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 20:09:06.298268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:06.295657 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-01f7d3-openshift-default-7bd4d4dd77-nhrrd"] Oct 30 20:09:07.496660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:07.496630 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:08.000427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:08.000395 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" path="/var/lib/kubelet/pods/cca576ba-0a1a-457c-be3a-c3bbaac08f0c/volumes" Oct 30 20:09:09.055768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:09.055734 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.193265 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:10.193231 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.326330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:10.326301 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.448020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:10.447935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.952370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:10.952332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:12.565554 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:12.565515 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:12.676385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:12.676299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:13.937386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:13.937350 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:13.953902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:13.953875 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:14.286298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:14.286228 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:09:14.755703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:14.755627 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:16.076443 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:09:16.076406 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:09:16.748675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:16.748643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.376258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:18.376224 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.500054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:18.500014 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.695135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:18.695097 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.806460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:18.806420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:20.574743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:20.574705 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:20.690136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:20.690097 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.496759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:22.496730 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:24.396803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:24.396764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:26.194454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:26.194419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:28.018480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:28.018445 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:29.835746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:29.835712 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:31.184593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:31.184567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:31.473344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:31.473263 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 20:09:31.473551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:31.473527 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" containerID="cri-o://59e75d016e09722c64b6f450a9deb67695ab2e8cc7cd153c8ed7c6eb676b254c" gracePeriod=30 Oct 30 20:09:31.596451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:31.596416 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:33.396847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:33.396815 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:35.223048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:35.223007 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:35.961532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:35.961481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:36.321261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:36.321185 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:09:36.330566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:36.330544 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:09:36.881638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:36.881567 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 20:09:36.949604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:36.949548 4959 scope.go:122] "RemoveContainer" containerID="59e75d016e09722c64b6f450a9deb67695ab2e8cc7cd153c8ed7c6eb676b254c" Oct 30 20:09:37.050589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050551 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.050765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050718 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.050840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050766 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.050840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050802 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.050840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050821 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.051005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050851 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.051005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.050950 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.051112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051016 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.051112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051029 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.051112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051070 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.051112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051100 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.051332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051142 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\") pod \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\" (UID: \"8a80b343-468d-46fe-b798-9bb1bcf76fd1\") " Oct 30 20:09:37.051332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051288 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.051440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051334 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:37.052051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051952 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.052051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051981 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.052051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.051998 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.052051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.052011 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.052051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.052025 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.053575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.053549 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:09:37.053820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.053792 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:09:37.054001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.053977 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "kube-api-access-tzwgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:37.054075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.054019 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token" pod "8a80b343-468d-46fe-b798-9bb1bcf76fd1" (UID: "8a80b343-468d-46fe-b798-9bb1bcf76fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:37.154755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.154728 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.154755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.154755 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.154755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.154768 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwgb\" (UniqueName: \"kubernetes.io/projected/8a80b343-468d-46fe-b798-9bb1bcf76fd1-kube-api-access-tzwgb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.155022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.154781 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b343-468d-46fe-b798-9bb1bcf76fd1-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:09:37.507104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.507027 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" Oct 30 20:09:37.507464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.507026 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" event={"ID":"8a80b343-468d-46fe-b798-9bb1bcf76fd1","Type":"ContainerDied","Data":"59e75d016e09722c64b6f450a9deb67695ab2e8cc7cd153c8ed7c6eb676b254c"} Oct 30 20:09:37.507464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.507161 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds" event={"ID":"8a80b343-468d-46fe-b798-9bb1bcf76fd1","Type":"ContainerDied","Data":"d8bbb2dec5f95d78dce29819cb01d8bdeea7b450a205bc9fa0ca0ab4840df0d8"} Oct 30 20:09:37.600061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.600030 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 20:09:37.607989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.607963 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a3426-openshift-default-6cb69b67b5-b6wds"] Oct 30 20:09:37.651627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:37.651601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.002337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:38.002301 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" path="/var/lib/kubelet/pods/8a80b343-468d-46fe-b798-9bb1bcf76fd1/volumes" Oct 30 20:09:39.494286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:39.494253 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:41.302808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:41.302774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:43.037383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:43.037356 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:44.791852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:44.791817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:46.067414 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:09:46.067371 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:09:46.908140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:46.904631 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:48.697636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:48.697604 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:50.616269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:50.615994 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:52.389203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:52.389169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:54.025363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:54.025326 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:55.914310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:55.914278 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:57.771935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:57.771892 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:58.035358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:58.035223 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:59.594727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:09:59.594694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:01.430710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:01.430668 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:03.303724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:03.303686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:05.203011 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:05.202952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:07.019937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:07.019897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:08.922719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:08.922685 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:10.721936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:10.721900 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:12.515051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:12.515002 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:14.305729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:14.305683 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:16.003149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:16.003111 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:16.127204 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:10:16.127174 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:10:17.801744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:17.801715 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.685430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:19.685397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:21.410459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:21.410422 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:23.298701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:23.298671 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.947699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:24.947656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:26.696561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:26.696530 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:28.732223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:28.732189 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:30.653522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:30.653448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.059307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:31.059212 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.336765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:31.336640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:32.572821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:32.572756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.529263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:33.529229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:34.667345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:34.667247 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:35.988306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:35.988270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.737608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:36.737578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:38.525370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:38.525327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:40.514083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:40.514045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:42.704528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:42.704474 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:44.710432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:44.710392 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:46.067533 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:10:46.067472 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:10:46.504017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:46.503989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:48.306456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:48.306417 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:50.082651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:50.082591 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:51.875321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:51.875289 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.337930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:53.337897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.444266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:53.444232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.695256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:53.695212 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:55.630189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:55.630157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:57.609906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:57.609870 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.492880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:10:59.492845 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:01.299164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:01.299133 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.193818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:03.193778 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:04.956951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:04.956917 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:06.808205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:06.808153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:08.581121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:08.581059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:10.394663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:10.394606 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.139092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:12.139059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:13.905776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:13.905744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:15.900670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:15.900586 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:16.068228 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:11:16.068195 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:11:16.277975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:16.277861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:17.903028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:17.902993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:19.683950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:19.683912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:21.625060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:21.625026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.607577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:23.607540 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.440218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:25.440174 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.278944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:27.278908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.982658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:27.982621 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:11:27.982849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:27.982688 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:11:29.030418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:29.030379 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:30.830952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:30.830923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.452052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.452017 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.674898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.674861 4959 generic.go:356] "Generic (PLEG): container finished" podID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerID="67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1" exitCode=0 Oct 30 20:11:31.675089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.674934 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerDied","Data":"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1"} Oct 30 20:11:31.675486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.675464 4959 scope.go:122] "RemoveContainer" containerID="67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1" Oct 30 20:11:31.677293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.677262 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:11:31.677411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.677347 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:11:31.677411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:31.677401 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:11:32.635384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:32.635344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:32.708020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:32.706657 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerStarted","Data":"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35"} Oct 30 20:11:33.472731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:33.472686 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:11:33.472964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:33.472911 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:11:33.473047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:33.472966 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 20:11:34.472898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:34.472855 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:11:34.473447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:34.472933 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 20:11:34.509722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:34.509692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.472978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:35.472936 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:11:35.473547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:35.473015 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 20:11:36.233557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:36.233487 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.472875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:36.472833 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:11:36.473055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:36.472885 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 20:11:37.471932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:37.471893 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 20:11:37.472242 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:37.472211 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 20:11:38.100394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:38.100345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:38.641381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:38.641349 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:11:38.642455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:38.642430 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:11:38.645150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:38.645119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:11:40.011969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:40.011934 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.711859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:41.711820 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.223925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:42.223890 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.369831 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:42.369796 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.417376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:43.417327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:45.472329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:45.472264 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:46.003808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:46.003777 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 20:11:46.003997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:46.003965 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" containerID="cri-o://90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f" gracePeriod=30 Oct 30 20:11:46.072482 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:11:46.072407 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:11:47.214598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:47.214564 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:49.096619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:49.096553 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.787557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:50.787522 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:51.155949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.155902 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:11:51.156968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.156935 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 20:11:51.317562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.317432 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 20:11:51.397942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.397906 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:51.462005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.461971 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462073 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462119 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462180 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462203 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462240 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462257 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:51.462525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462377 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:51.462525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462390 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462428 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.462525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462473 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\") pod \"8be64621-aa39-4baa-aad0-c669440ee371\" (UID: \"8be64621-aa39-4baa-aad0-c669440ee371\") " Oct 30 20:11:51.463063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462597 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:51.463063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462730 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:51.463063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.462924 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:51.463810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.463785 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.463931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.463816 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.463931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.463831 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8be64621-aa39-4baa-aad0-c669440ee371-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.463931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.463844 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.463931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.463858 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.464657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.464631 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:51.464895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.464874 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:51.464895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.464886 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "kube-api-access-lhx5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:51.465023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.464938 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token" pod "8be64621-aa39-4baa-aad0-c669440ee371" (UID: "8be64621-aa39-4baa-aad0-c669440ee371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:51.566523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.566149 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.566523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.566185 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8be64621-aa39-4baa-aad0-c669440ee371-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.566523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.566199 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhx5r\" (UniqueName: \"kubernetes.io/projected/8be64621-aa39-4baa-aad0-c669440ee371-kube-api-access-lhx5r\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.566523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.566212 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8be64621-aa39-4baa-aad0-c669440ee371-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:11:51.929814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.929776 4959 generic.go:356] "Generic (PLEG): container finished" podID="8be64621-aa39-4baa-aad0-c669440ee371" containerID="90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f" exitCode=0 Oct 30 20:11:51.930299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.929845 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" event={"ID":"8be64621-aa39-4baa-aad0-c669440ee371","Type":"ContainerDied","Data":"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f"} Oct 30 20:11:51.930299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.929914 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" event={"ID":"8be64621-aa39-4baa-aad0-c669440ee371","Type":"ContainerDied","Data":"6a824feb4f9e8424f4fd78a1d1cdbf5106d3382dcbafab02cdc7ad53804f977d"} Oct 30 20:11:51.930299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.929934 4959 scope.go:122] "RemoveContainer" containerID="90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f" Oct 30 20:11:51.930299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.929916 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc" Oct 30 20:11:51.944798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.944775 4959 scope.go:122] "RemoveContainer" containerID="90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f" Oct 30 20:11:51.945250 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:11:51.945224 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f\": container with ID starting with 90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f not found: ID does not exist" containerID="90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f" Oct 30 20:11:51.945354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:51.945260 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f"} err="failed to get container status \"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f\": rpc error: code = NotFound desc = could not find container \"90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f\": container with ID starting with 90ebddc3f346b066664850558be3d1c59e147f4650359bbdaae8e4f618fb744f not found: ID does not exist" Oct 30 20:11:52.002443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:52.002411 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 20:11:52.006062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:52.006035 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-284bf6-openshift-default-59b45b4cf7-2gtpc"] Oct 30 20:11:53.039191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:53.039160 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:54.000020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:53.999986 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be64621-aa39-4baa-aad0-c669440ee371" path="/var/lib/kubelet/pods/8be64621-aa39-4baa-aad0-c669440ee371/volumes" Oct 30 20:11:55.007145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:55.007113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.004777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:57.004736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:58.916037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:11:58.916006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:00.916941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:00.916911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.162472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:02.162423 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.281532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:02.272075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:03.195153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:03.195116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.095964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:05.095930 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.911098 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:06.911070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:08.891344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:08.891309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.853100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:10.853030 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.607981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:12.607950 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:14.551940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:14.543199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.068429 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:12:16.068391 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:12:16.409723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:16.409690 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.195607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:17.195576 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:12:17.196034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:17.195791 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" containerID="cri-o://46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df" gracePeriod=30 Oct 30 20:12:18.224095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:18.224056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.113811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:20.113776 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.270417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.270372 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.618745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.618714 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:12:22.709295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.709255 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.709513 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.709359 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.709915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.709860 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:22.710087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.709929 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:22.710087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.709948 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710020 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710107 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710150 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710206 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710259 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710372 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\") pod \"41c3ac53-79b3-4839-8679-7bea0f99d22a\" (UID: \"41c3ac53-79b3-4839-8679-7bea0f99d22a\") " Oct 30 20:12:22.710595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.710432 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:22.711251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.711195 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.711401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.711387 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.711528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.711514 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.711628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.711385 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:22.711787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.711753 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:22.717708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.717637 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:22.717916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.717677 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:22.718024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.717746 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "kube-api-access-brfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:22.740929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.740886 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo" pod "41c3ac53-79b3-4839-8679-7bea0f99d22a" (UID: "41c3ac53-79b3-4839-8679-7bea0f99d22a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:22.813506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813468 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfhn\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-kube-api-access-brfhn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.813701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813518 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.813701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813534 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.813701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813547 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c3ac53-79b3-4839-8679-7bea0f99d22a-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.813701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813560 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c3ac53-79b3-4839-8679-7bea0f99d22a-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:22.813701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:22.813574 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c3ac53-79b3-4839-8679-7bea0f99d22a-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:23.251001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.250951 4959 generic.go:356] "Generic (PLEG): container finished" podID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerID="46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df" exitCode=0 Oct 30 20:12:23.251267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.251042 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" Oct 30 20:12:23.251267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.251039 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" event={"ID":"41c3ac53-79b3-4839-8679-7bea0f99d22a","Type":"ContainerDied","Data":"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df"} Oct 30 20:12:23.251267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.251155 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd" event={"ID":"41c3ac53-79b3-4839-8679-7bea0f99d22a","Type":"ContainerDied","Data":"f70011e73b9aace9a62ed251a7ba49bdd49428b3224eccfe8fec80833d53f58d"} Oct 30 20:12:23.251267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.251195 4959 scope.go:122] "RemoveContainer" containerID="46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df" Oct 30 20:12:23.269324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.269300 4959 scope.go:122] "RemoveContainer" containerID="46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df" Oct 30 20:12:23.269746 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:12:23.269720 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df\": container with ID starting with 46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df not found: ID does not exist" containerID="46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df" Oct 30 20:12:23.269863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.269755 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df"} err="failed to get container status \"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df\": rpc error: code = NotFound desc = could not find container \"46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df\": container with ID starting with 46e7612845a0db79efa72842abcfe0bbd191c88b9f44c0f0c66f083d4729a9df not found: ID does not exist" Oct 30 20:12:23.304651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.304616 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:12:23.315582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:23.315551 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-sqznd"] Oct 30 20:12:24.006077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:24.006043 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" path="/var/lib/kubelet/pods/41c3ac53-79b3-4839-8679-7bea0f99d22a/volumes" Oct 30 20:12:24.112175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:24.112137 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.937668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:25.937630 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.703724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:27.703694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.448679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:29.448640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.676833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:29.676798 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.539439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:31.539407 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.760206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:31.760172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:32.133408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:32.133375 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:33.186654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:33.186618 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.986990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:34.986954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:36.795222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:36.795179 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:38.642395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:38.642360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.324148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:40.324115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:42.032410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:42.032374 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:43.802952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:43.802915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.624827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:45.624789 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:46.067419 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:12:46.067382 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:12:47.452099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:47.452069 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:49.112178 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:49.112139 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:50.828846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:50.828790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.221623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.221589 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:12:52.222083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.221834 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" podUID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" containerName="maas-api" containerID="cri-o://a721002f79563d4540ab7d1ec15e603f1037fdce7178c268f3211ea30249b650" gracePeriod=30 Oct 30 20:12:52.287871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.287834 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.596725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.596652 4959 generic.go:356] "Generic (PLEG): container finished" podID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" containerID="a721002f79563d4540ab7d1ec15e603f1037fdce7178c268f3211ea30249b650" exitCode=0 Oct 30 20:12:52.597023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.596722 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" event={"ID":"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac","Type":"ContainerDied","Data":"a721002f79563d4540ab7d1ec15e603f1037fdce7178c268f3211ea30249b650"} Oct 30 20:12:52.664522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.647403 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:12:52.771126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.769685 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\") pod \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " Oct 30 20:12:52.771126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.769922 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\" (UniqueName: \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\") pod \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " Oct 30 20:12:52.771126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.769967 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\") pod \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " Oct 30 20:12:52.771126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.770012 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\") pod \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\" (UID: \"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac\") " Oct 30 20:12:52.772651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.772359 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca" pod "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" (UID: "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:52.810773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.808718 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls" pod "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" (UID: "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:52.810773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.810070 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs" pod "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" (UID: "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:52.810773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.810233 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57" pod "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" (UID: "3d4f37f9-488a-4f2e-a27c-a96b9033d2ac"). InnerVolumeSpecName "kube-api-access-x8c57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:52.872979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.872888 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8c57\" (UniqueName: \"kubernetes.io/projected/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-kube-api-access-x8c57\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:52.872979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.872931 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:52.872979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.872947 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:52.872979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:52.872960 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:12:53.140091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.140061 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.355952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.355921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.525802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.525719 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.634038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.633990 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" event={"ID":"3d4f37f9-488a-4f2e-a27c-a96b9033d2ac","Type":"ContainerDied","Data":"d95f23e9633e2055a302bc29ae4d2adc2581f578f3f7fc0845c34e24d468f7de"} Oct 30 20:12:53.634376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.634356 4959 scope.go:122] "RemoveContainer" containerID="a721002f79563d4540ab7d1ec15e603f1037fdce7178c268f3211ea30249b650" Oct 30 20:12:53.634560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.634218 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56" Oct 30 20:12:53.685422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.685385 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:12:53.705312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.705274 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ef132f-78bb74786c-6ns56"] Oct 30 20:12:53.920943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:53.920904 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:54.002683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:54.002650 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" path="/var/lib/kubelet/pods/3d4f37f9-488a-4f2e-a27c-a96b9033d2ac/volumes" Oct 30 20:12:55.363071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:55.363037 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.005258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:57.005226 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:58.782615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:12:58.782578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.317190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:00.317155 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.426885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:00.426850 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.728988 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:00.728949 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.891631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:00.882811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:02.595847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:02.595813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:04.479802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:04.479772 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:06.218380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:06.218346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:08.091282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:08.091244 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.812962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:09.812928 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.403923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:11.403891 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:13:11.440759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:11.440719 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:13:11.488680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:11.488652 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.597020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:11.596978 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.229544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:13.229510 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:14.908447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:14.908413 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:16.069418 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:13:16.069382 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:13:16.405210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.405169 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" podUID="e99fd189-0734-45dc-ac8a-f7f526288126" containerName="payload-processing" containerID="cri-o://bbaae0d3120ad307c9b0ad5cf12ac3b121c6ffa08bceae285ea6777bba01ca7b" gracePeriod=25 Oct 30 20:13:16.441123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.441075 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" containerID="cri-o://ab98eadada824542133f092e3dd26cb06204ec4d18b70a599b6c4714cc98d1f3" gracePeriod=25 Oct 30 20:13:16.601158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.600126 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:16.885196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.885153 4959 generic.go:356] "Generic (PLEG): container finished" podID="e99fd189-0734-45dc-ac8a-f7f526288126" containerID="bbaae0d3120ad307c9b0ad5cf12ac3b121c6ffa08bceae285ea6777bba01ca7b" exitCode=0 Oct 30 20:13:16.885428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.885324 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" event={"ID":"e99fd189-0734-45dc-ac8a-f7f526288126","Type":"ContainerDied","Data":"bbaae0d3120ad307c9b0ad5cf12ac3b121c6ffa08bceae285ea6777bba01ca7b"} Oct 30 20:13:16.922585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:16.922558 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:13:17.024526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.002152 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\" (UniqueName: \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\") pod \"e99fd189-0734-45dc-ac8a-f7f526288126\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " Oct 30 20:13:17.025008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.024965 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\") pod \"e99fd189-0734-45dc-ac8a-f7f526288126\" (UID: \"e99fd189-0734-45dc-ac8a-f7f526288126\") " Oct 30 20:13:17.033834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.033787 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume" pod "e99fd189-0734-45dc-ac8a-f7f526288126" (UID: "e99fd189-0734-45dc-ac8a-f7f526288126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:17.121568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.098706 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:13:17.135530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.132312 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99fd189-0734-45dc-ac8a-f7f526288126-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:17.198890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.198837 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf" pod "e99fd189-0734-45dc-ac8a-f7f526288126" (UID: "e99fd189-0734-45dc-ac8a-f7f526288126"). InnerVolumeSpecName "kube-api-access-hz7lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:17.234074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.234036 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7lf\" (UniqueName: \"kubernetes.io/projected/e99fd189-0734-45dc-ac8a-f7f526288126-kube-api-access-hz7lf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:17.897737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.897708 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" Oct 30 20:13:17.897928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.897708 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt" event={"ID":"e99fd189-0734-45dc-ac8a-f7f526288126","Type":"ContainerDied","Data":"c138235bdbf9bf2e301979914e10742296e27affea7fac0d2faedf44ac76349e"} Oct 30 20:13:17.897928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.897846 4959 scope.go:122] "RemoveContainer" containerID="bbaae0d3120ad307c9b0ad5cf12ac3b121c6ffa08bceae285ea6777bba01ca7b" Oct 30 20:13:17.954851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.954813 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:13:17.959105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:17.959063 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-936f70-78f8989876-wmljt"] Oct 30 20:13:18.004203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:18.004167 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99fd189-0734-45dc-ac8a-f7f526288126" path="/var/lib/kubelet/pods/e99fd189-0734-45dc-ac8a-f7f526288126/volumes" Oct 30 20:13:18.742294 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:18.742256 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.497008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:20.496971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.247971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:22.247933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:23.870931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:23.870899 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:25.512969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:25.512933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:25.622976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:25.622939 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.078920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:27.078879 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:13:27.282843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:27.282809 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:29.033694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:29.033658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:30.686245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:30.686210 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:31.725081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:31.725045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.147790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.147755 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.426329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.426239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.476806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.476758 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:13:32.477415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.477377 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" containerID="cri-o://13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f" gracePeriod=30 Oct 30 20:13:32.544538 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.544473 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:13:32.544980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:32.544942 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" containerID="cri-o://2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc" gracePeriod=30 Oct 30 20:13:34.202874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:34.202785 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:35.932004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:35.931968 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:36.750890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.750858 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:13:36.934963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.934932 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\") pod \"ad91e702-e7fc-4865-9ae6-1e61b631b593\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " Oct 30 20:13:36.935467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.935015 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\" (UniqueName: \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\") pod \"ad91e702-e7fc-4865-9ae6-1e61b631b593\" (UID: \"ad91e702-e7fc-4865-9ae6-1e61b631b593\") " Oct 30 20:13:36.935467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.935384 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume" pod "ad91e702-e7fc-4865-9ae6-1e61b631b593" (UID: "ad91e702-e7fc-4865-9ae6-1e61b631b593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:36.935882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.935861 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad91e702-e7fc-4865-9ae6-1e61b631b593-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:36.937356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:36.937335 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54" pod "ad91e702-e7fc-4865-9ae6-1e61b631b593" (UID: "ad91e702-e7fc-4865-9ae6-1e61b631b593"). InnerVolumeSpecName "kube-api-access-s5w54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:37.036936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.036846 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5w54\" (UniqueName: \"kubernetes.io/projected/ad91e702-e7fc-4865-9ae6-1e61b631b593-kube-api-access-s5w54\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:37.057458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.057423 4959 scope.go:122] "RemoveContainer" containerID="ab98eadada824542133f092e3dd26cb06204ec4d18b70a599b6c4714cc98d1f3" Oct 30 20:13:37.071855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.071827 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" Oct 30 20:13:37.071978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.071852 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" event={"ID":"ad91e702-e7fc-4865-9ae6-1e61b631b593","Type":"ContainerDied","Data":"ab98eadada824542133f092e3dd26cb06204ec4d18b70a599b6c4714cc98d1f3"} Oct 30 20:13:37.071978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.071934 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr" event={"ID":"ad91e702-e7fc-4865-9ae6-1e61b631b593","Type":"ContainerDied","Data":"c1692488da5d73d00dfaa82d84ee4bef70fa2ef6ac68fb4252173420daa10d4e"} Oct 30 20:13:37.124132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.124103 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:13:37.133734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.133705 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-936f70-6c4b6fd79-g2qjr"] Oct 30 20:13:37.658552 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.658520 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.942892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.942866 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:13:37.950619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:37.950598 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:13:38.001224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.001192 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" path="/var/lib/kubelet/pods/ad91e702-e7fc-4865-9ae6-1e61b631b593/volumes" Oct 30 20:13:38.054093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.054058 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.054390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.054366 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.054594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.054575 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.054739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.054721 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.054875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.054860 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.055049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055033 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.055173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055158 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.055418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055399 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.055594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055577 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.055748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055731 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.055873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055855 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.055984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.055970 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.056118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056103 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.056252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056238 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.056384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056367 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.056535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056519 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\") pod \"cae28612-2090-4280-9139-63ef293ba821\" (UID: \"cae28612-2090-4280-9139-63ef293ba821\") " Oct 30 20:13:38.056686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056670 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.056859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.056842 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\") pod \"440c558d-c52a-45f1-964e-bc090bd0268c\" (UID: \"440c558d-c52a-45f1-964e-bc090bd0268c\") " Oct 30 20:13:38.058153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.058128 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.058539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.058471 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.058788 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.058767 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.059676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.059633 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.059877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.059854 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.061082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.060890 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.061685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.061399 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:38.061871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.061843 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.062094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.062071 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.062442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.062413 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:38.062832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.062803 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:38.065878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.065852 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:38.065878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.065860 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.066051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.065878 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:38.066051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.065978 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:38.066051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.066013 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd" pod "cae28612-2090-4280-9139-63ef293ba821" (UID: "cae28612-2090-4280-9139-63ef293ba821"). InnerVolumeSpecName "kube-api-access-rrwnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:38.066222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.066060 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "kube-api-access-frnvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:38.067997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.067970 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy" pod "440c558d-c52a-45f1-964e-bc090bd0268c" (UID: "440c558d-c52a-45f1-964e-bc090bd0268c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:38.080924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.080893 4959 generic.go:356] "Generic (PLEG): container finished" podID="cae28612-2090-4280-9139-63ef293ba821" containerID="2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc" exitCode=0 Oct 30 20:13:38.081056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.080944 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" Oct 30 20:13:38.081056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.080974 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" event={"ID":"cae28612-2090-4280-9139-63ef293ba821","Type":"ContainerDied","Data":"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc"} Oct 30 20:13:38.081056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.081024 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp" event={"ID":"cae28612-2090-4280-9139-63ef293ba821","Type":"ContainerDied","Data":"7e3c85511d024d957b27f7b9a533e9f9c7b5ddc0a76e80eb87444ca97b911e8a"} Oct 30 20:13:38.081202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.081056 4959 scope.go:122] "RemoveContainer" containerID="2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc" Oct 30 20:13:38.083000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.082978 4959 generic.go:356] "Generic (PLEG): container finished" podID="440c558d-c52a-45f1-964e-bc090bd0268c" containerID="13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f" exitCode=0 Oct 30 20:13:38.083078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.083032 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" event={"ID":"440c558d-c52a-45f1-964e-bc090bd0268c","Type":"ContainerDied","Data":"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f"} Oct 30 20:13:38.083078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.083072 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" event={"ID":"440c558d-c52a-45f1-964e-bc090bd0268c","Type":"ContainerDied","Data":"deba3a06d4c5f61f243a7d3ffe37e8bffe380b111572c3290fb658c9040d8c21"} Oct 30 20:13:38.083181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.083139 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq" Oct 30 20:13:38.096646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.096623 4959 scope.go:122] "RemoveContainer" containerID="2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc" Oct 30 20:13:38.098370 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:13:38.098341 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc\": container with ID starting with 2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc not found: ID does not exist" containerID="2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc" Oct 30 20:13:38.098456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.098382 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc"} err="failed to get container status \"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc\": rpc error: code = NotFound desc = could not find container \"2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc\": container with ID starting with 2d26790b17af0f0505915d47002065bb1cab70acae9fbf487be4f10fe46091dc not found: ID does not exist" Oct 30 20:13:38.098456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.098406 4959 scope.go:122] "RemoveContainer" containerID="13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f" Oct 30 20:13:38.110369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.110349 4959 scope.go:122] "RemoveContainer" containerID="13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f" Oct 30 20:13:38.110707 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:13:38.110685 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f\": container with ID starting with 13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f not found: ID does not exist" containerID="13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f" Oct 30 20:13:38.110811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.110721 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f"} err="failed to get container status \"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f\": rpc error: code = NotFound desc = could not find container \"13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f\": container with ID starting with 13043cabc6f1c222cc7ef5f7c59642a9bc1d5af363329e469ed54ce17947e28f not found: ID does not exist" Oct 30 20:13:38.133782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.133760 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:13:38.137799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.137774 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-29rjp"] Oct 30 20:13:38.159407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.159375 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.159909 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.159892 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae28612-2090-4280-9139-63ef293ba821-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160028 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160052 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160068 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwnd\" (UniqueName: \"kubernetes.io/projected/cae28612-2090-4280-9139-63ef293ba821-kube-api-access-rrwnd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160083 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440c558d-c52a-45f1-964e-bc090bd0268c-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160097 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160183 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160198 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160212 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae28612-2090-4280-9139-63ef293ba821-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160224 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160237 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440c558d-c52a-45f1-964e-bc090bd0268c-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160250 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160265 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae28612-2090-4280-9139-63ef293ba821-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160278 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160292 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnvr\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-kube-api-access-frnvr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160308 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440c558d-c52a-45f1-964e-bc090bd0268c-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.160402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.160320 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440c558d-c52a-45f1-964e-bc090bd0268c-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:13:38.183303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.183275 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:13:38.187298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:38.187275 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cd2ad-openshift-default-79c44c78df-p9sxq"] Oct 30 20:13:39.418362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:39.418318 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:40.001447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:40.001405 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" path="/var/lib/kubelet/pods/440c558d-c52a-45f1-964e-bc090bd0268c/volumes" Oct 30 20:13:40.002236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:40.002208 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae28612-2090-4280-9139-63ef293ba821" path="/var/lib/kubelet/pods/cae28612-2090-4280-9139-63ef293ba821/volumes" Oct 30 20:13:41.173753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:41.173682 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:42.897479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:42.897404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:44.616937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:44.616902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:46.068723 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:13:46.068684 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:13:46.567565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:46.564133 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:47.294732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:47.294694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:48.487790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:48.487757 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:50.157284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:50.157252 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:51.529257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:51.529224 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:51.640186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:51.640108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.527890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:52.527857 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.711984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:52.711944 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.826574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:52.826456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.284235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:53.284200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:54.635327 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:54.635295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:56.494606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:56.494549 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:57.218805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.218770 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 20:13:57.219001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.218959 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" containerID="cri-o://2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c" gracePeriod=30 Oct 30 20:13:57.230681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.230655 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 20:13:57.230917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.230891 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" containerID="cri-o://630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45" gracePeriod=30 Oct 30 20:13:57.234410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.234385 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 20:13:57.234638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.234616 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" containerID="cri-o://4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a" gracePeriod=30 Oct 30 20:13:57.242858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.242833 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 20:13:57.243023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:57.243003 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" containerID="cri-o://a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9" gracePeriod=30 Oct 30 20:13:58.193468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:58.193426 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.329687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:59.329649 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.440171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:59.440141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.671339 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:13:59.671307 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:00.186931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:00.186889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:00.760517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:00.760468 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:14:00.806312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:00.806276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.093112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.093081 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.565946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.565553 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 20:14:02.589873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.589845 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 20:14:02.677315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677287 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677344 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677366 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677402 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677434 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677484 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677548 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677623 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677635 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677692 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677766 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677777 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677796 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677864 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.677868 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678040 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.678104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678088 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678134 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678173 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678201 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\") pod \"8af5edd2-6de4-49c1-b711-48d353cab69d\" (UID: \"8af5edd2-6de4-49c1-b711-48d353cab69d\") " Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678245 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\") pod \"d0c07e0b-d292-4832-81c2-d5198affed5b\" (UID: \"d0c07e0b-d292-4832-81c2-d5198affed5b\") " Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.678758 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679024 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679597 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679621 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679635 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679650 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.679687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.679673 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.680207 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.680851 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.680920 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.681217 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.681517 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.682223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.681711 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.683116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.683081 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "kube-api-access-zd6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.684124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.683991 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:02.684124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.684072 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.684905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.684875 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.685158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.685136 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "kube-api-access-fq2hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.685444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.685425 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token" pod "8af5edd2-6de4-49c1-b711-48d353cab69d" (UID: "8af5edd2-6de4-49c1-b711-48d353cab69d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.686265 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.686226 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token" pod "d0c07e0b-d292-4832-81c2-d5198affed5b" (UID: "d0c07e0b-d292-4832-81c2-d5198affed5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.694909 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.694598 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 20:14:02.696101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.696079 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.701406 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.701334 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 20:14:02.780737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780706 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.780931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780770 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.780931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780814 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.780931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780854 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.780931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780904 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.781173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780942 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.781173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.780978 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.781173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.781068 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.781173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.781138 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.781480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.781450 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.782307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.782276 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.782410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.782372 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.782783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.782760 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.783021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.782988 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.783131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783116 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.783193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783162 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.783193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783188 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.783311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783255 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.783311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783287 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.783429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783324 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.783429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783363 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\") pod \"b9a6dc58-5919-49b9-9165-466126251f9a\" (UID: \"b9a6dc58-5919-49b9-9165-466126251f9a\") " Oct 30 20:14:02.783564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783457 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\") pod \"e3c04781-2619-4bc2-af4e-6c873bd49547\" (UID: \"e3c04781-2619-4bc2-af4e-6c873bd49547\") " Oct 30 20:14:02.783625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783576 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:02.783803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783750 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.783904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783882 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.784017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.783996 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.784115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.784089 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.784240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.784185 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.784315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.784294 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.787108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.787064 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.787226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.787187 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:02.789739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789716 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789744 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789758 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789773 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c07e0b-d292-4832-81c2-d5198affed5b-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789787 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a6dc58-5919-49b9-9165-466126251f9a-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789799 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a6dc58-5919-49b9-9165-466126251f9a-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789812 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.789832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789825 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789838 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789851 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789864 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789878 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af5edd2-6de4-49c1-b711-48d353cab69d-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789891 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789904 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789919 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789932 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789945 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2hd\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-kube-api-access-fq2hd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789960 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789973 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd6pw\" (UniqueName: \"kubernetes.io/projected/d0c07e0b-d292-4832-81c2-d5198affed5b-kube-api-access-zd6pw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.789989 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c04781-2619-4bc2-af4e-6c873bd49547-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790002 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790014 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790025 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790038 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af5edd2-6de4-49c1-b711-48d353cab69d-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790050 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a6dc58-5919-49b9-9165-466126251f9a-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.790131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.790063 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c07e0b-d292-4832-81c2-d5198affed5b-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.793347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.793323 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:02.793453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.793374 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.793688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.793668 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "kube-api-access-dnx59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.794006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.793990 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy" pod "e3c04781-2619-4bc2-af4e-6c873bd49547" (UID: "e3c04781-2619-4bc2-af4e-6c873bd49547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:02.795154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.795136 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8" pod "b9a6dc58-5919-49b9-9165-466126251f9a" (UID: "b9a6dc58-5919-49b9-9165-466126251f9a"). InnerVolumeSpecName "kube-api-access-xpnp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:02.891059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.891026 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.891059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.891058 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.891059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.891071 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c04781-2619-4bc2-af4e-6c873bd49547-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.891315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.891084 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnx59\" (UniqueName: \"kubernetes.io/projected/e3c04781-2619-4bc2-af4e-6c873bd49547-kube-api-access-dnx59\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:02.891315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:02.891098 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpnp8\" (UniqueName: \"kubernetes.io/projected/b9a6dc58-5919-49b9-9165-466126251f9a-kube-api-access-xpnp8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:03.346169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.346126 4959 generic.go:356] "Generic (PLEG): container finished" podID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerID="2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c" exitCode=0 Oct 30 20:14:03.346672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.346177 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" Oct 30 20:14:03.346672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.346208 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" event={"ID":"d0c07e0b-d292-4832-81c2-d5198affed5b","Type":"ContainerDied","Data":"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c"} Oct 30 20:14:03.346672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.346275 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx" event={"ID":"d0c07e0b-d292-4832-81c2-d5198affed5b","Type":"ContainerDied","Data":"cbdbf3c9bc3700f0e409bfa43974c8b91f5625799c7bd08fcc175a903be733ab"} Oct 30 20:14:03.346672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.346297 4959 scope.go:122] "RemoveContainer" containerID="2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c" Oct 30 20:14:03.348132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.348103 4959 generic.go:356] "Generic (PLEG): container finished" podID="b9a6dc58-5919-49b9-9165-466126251f9a" containerID="a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9" exitCode=0 Oct 30 20:14:03.348259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.348137 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" event={"ID":"b9a6dc58-5919-49b9-9165-466126251f9a","Type":"ContainerDied","Data":"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9"} Oct 30 20:14:03.348259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.348170 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" Oct 30 20:14:03.348259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.348217 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d" event={"ID":"b9a6dc58-5919-49b9-9165-466126251f9a","Type":"ContainerDied","Data":"4bdee8b74be08c03a1b55987e26aab3a2a27a5c1ebe3fef9412004bc37134177"} Oct 30 20:14:03.352859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.352831 4959 generic.go:356] "Generic (PLEG): container finished" podID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerID="630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45" exitCode=0 Oct 30 20:14:03.353005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.352884 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" Oct 30 20:14:03.353005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.352924 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" event={"ID":"8af5edd2-6de4-49c1-b711-48d353cab69d","Type":"ContainerDied","Data":"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45"} Oct 30 20:14:03.353005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.352969 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx" event={"ID":"8af5edd2-6de4-49c1-b711-48d353cab69d","Type":"ContainerDied","Data":"fa882d34cf9c479dfe8151aec9fa5db5a3b6c39d1f5799bf46f086f657ffa70e"} Oct 30 20:14:03.354315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.354294 4959 generic.go:356] "Generic (PLEG): container finished" podID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerID="4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a" exitCode=0 Oct 30 20:14:03.354416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.354339 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" Oct 30 20:14:03.354416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.354352 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" event={"ID":"e3c04781-2619-4bc2-af4e-6c873bd49547","Type":"ContainerDied","Data":"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a"} Oct 30 20:14:03.354416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.354393 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc" event={"ID":"e3c04781-2619-4bc2-af4e-6c873bd49547","Type":"ContainerDied","Data":"c654f0f934a90fd328ab9739463b7cc8e26b218fb44e380e0f38f3e98eaddfba"} Oct 30 20:14:03.361755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.361556 4959 scope.go:122] "RemoveContainer" containerID="2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c" Oct 30 20:14:03.362024 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:03.361999 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c\": container with ID starting with 2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c not found: ID does not exist" containerID="2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c" Oct 30 20:14:03.362093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.362038 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c"} err="failed to get container status \"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c\": rpc error: code = NotFound desc = could not find container \"2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c\": container with ID starting with 2cdd65bda774daa9f9f1ea3b272d1a9f32fcfbce0b6831520edc51c7c3c4573c not found: ID does not exist" Oct 30 20:14:03.362093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.362066 4959 scope.go:122] "RemoveContainer" containerID="a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9" Oct 30 20:14:03.381853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.381826 4959 scope.go:122] "RemoveContainer" containerID="a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9" Oct 30 20:14:03.382196 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:03.382178 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9\": container with ID starting with a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9 not found: ID does not exist" containerID="a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9" Oct 30 20:14:03.382255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.382207 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9"} err="failed to get container status \"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9\": rpc error: code = NotFound desc = could not find container \"a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9\": container with ID starting with a12512702471384d85d882d898106b52effef33e44d987ce045dda8adde6d2d9 not found: ID does not exist" Oct 30 20:14:03.382255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.382225 4959 scope.go:122] "RemoveContainer" containerID="630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45" Oct 30 20:14:03.394333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.394310 4959 scope.go:122] "RemoveContainer" containerID="630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45" Oct 30 20:14:03.394669 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:03.394639 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45\": container with ID starting with 630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45 not found: ID does not exist" containerID="630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45" Oct 30 20:14:03.394790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.394675 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45"} err="failed to get container status \"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45\": rpc error: code = NotFound desc = could not find container \"630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45\": container with ID starting with 630ac32c4d2f10e318ce25ac48c06690b5d5db2de9fefe59967c44871f272b45 not found: ID does not exist" Oct 30 20:14:03.394790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.394692 4959 scope.go:122] "RemoveContainer" containerID="4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a" Oct 30 20:14:03.406164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.406142 4959 scope.go:122] "RemoveContainer" containerID="4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a" Oct 30 20:14:03.406542 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:03.406481 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a\": container with ID starting with 4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a not found: ID does not exist" containerID="4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a" Oct 30 20:14:03.406614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.406555 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a"} err="failed to get container status \"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a\": rpc error: code = NotFound desc = could not find container \"4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a\": container with ID starting with 4b33d47fa8d6ddae73deefa03eb21b6cbc678935adf560dbad313673c55a5f3a not found: ID does not exist" Oct 30 20:14:03.412837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.412813 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 20:14:03.423038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.423007 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-42srx"] Oct 30 20:14:03.474729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.474700 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 20:14:03.480666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.480643 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lr88d"] Oct 30 20:14:03.513753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.513678 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 20:14:03.517120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.517103 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-gqnwx"] Oct 30 20:14:03.550067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.550040 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 20:14:03.553613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:03.553578 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-fh6gc"] Oct 30 20:14:04.000183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:04.000150 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" path="/var/lib/kubelet/pods/8af5edd2-6de4-49c1-b711-48d353cab69d/volumes" Oct 30 20:14:04.000714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:04.000697 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" path="/var/lib/kubelet/pods/b9a6dc58-5919-49b9-9165-466126251f9a/volumes" Oct 30 20:14:04.001245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:04.001228 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" path="/var/lib/kubelet/pods/d0c07e0b-d292-4832-81c2-d5198affed5b/volumes" Oct 30 20:14:04.001996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:04.001980 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" path="/var/lib/kubelet/pods/e3c04781-2619-4bc2-af4e-6c873bd49547/volumes" Oct 30 20:14:04.131472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:04.131442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.983887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:05.983852 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.704619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:07.704588 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:09.345279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:09.345236 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:11.111867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:11.111833 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.847545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:12.847508 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:14.569755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:14.569718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.796437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:15.796396 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" containerID="cri-o://7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c" gracePeriod=45 Oct 30 20:14:15.898648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:15.898604 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": dial tcp 10.132.0.161:8000: connect: connection refused" Oct 30 20:14:16.003976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.003943 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:14:16.035976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.035870 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:16.070373 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:16.070279 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:14:16.172528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.172480 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:14:16.321991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.321880 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.322150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322098 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.322200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322150 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\" (UniqueName: \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.322241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322232 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322428 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322561 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\") pod \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\" (UID: \"feb1a303-3f04-46c3-aa38-7ba67a642cfd\") " Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322561 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322612 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.322742 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.323124 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.323143 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.323173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.323153 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.325666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.325637 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "kube-api-access-zncj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:16.325917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.325896 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:16.327479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.327453 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs" pod "feb1a303-3f04-46c3-aa38-7ba67a642cfd" (UID: "feb1a303-3f04-46c3-aa38-7ba67a642cfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:16.424857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.424824 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb1a303-3f04-46c3-aa38-7ba67a642cfd-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.424857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.424852 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zncj7\" (UniqueName: \"kubernetes.io/projected/feb1a303-3f04-46c3-aa38-7ba67a642cfd-kube-api-access-zncj7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.424857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.424863 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb1a303-3f04-46c3-aa38-7ba67a642cfd-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:16.470041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.470003 4959 generic.go:356] "Generic (PLEG): container finished" podID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerID="7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c" exitCode=0 Oct 30 20:14:16.470234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.470049 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" Oct 30 20:14:16.470234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.470078 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" event={"ID":"feb1a303-3f04-46c3-aa38-7ba67a642cfd","Type":"ContainerDied","Data":"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c"} Oct 30 20:14:16.470234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.470125 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7" event={"ID":"feb1a303-3f04-46c3-aa38-7ba67a642cfd","Type":"ContainerDied","Data":"c2d8b8f7e8e87d956e7956b6202f5cd86ed34e9795b1c89418ad2049ff435ad6"} Oct 30 20:14:16.470234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.470144 4959 scope.go:122] "RemoveContainer" containerID="7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c" Oct 30 20:14:16.483176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.483158 4959 scope.go:122] "RemoveContainer" containerID="7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c" Oct 30 20:14:16.483468 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:16.483432 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c\": container with ID starting with 7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c not found: ID does not exist" containerID="7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c" Oct 30 20:14:16.483607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.483475 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c"} err="failed to get container status \"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c\": rpc error: code = NotFound desc = could not find container \"7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c\": container with ID starting with 7cb716d44179be8a025fcd5c1211591771a4298e4f34c97b80d797d6338d3f4c not found: ID does not exist" Oct 30 20:14:16.522367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.522331 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:14:16.528451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.528423 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c17154/test-model-c17154-kserve-7658cc6cb5-cp8j7"] Oct 30 20:14:16.586521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:16.586426 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:18.000059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:18.000027 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" path="/var/lib/kubelet/pods/feb1a303-3f04-46c3-aa38-7ba67a642cfd/volumes" Oct 30 20:14:18.552049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:18.551999 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:19.183802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:19.183767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:20.565666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:20.565617 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:21.413826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:21.413794 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:23.217336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:23.217297 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:25.097869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:25.097838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:26.682511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:26.682462 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:26.786485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:26.786434 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:28.430548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:28.430517 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:29.692851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:29.692820 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:14:29.832214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:29.832150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:29.948465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:29.948358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:30.134877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:30.134843 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.029427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.029385 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" containerName="main" containerID="cri-o://b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2" gracePeriod=45 Oct 30 20:14:31.316298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.316274 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:14:31.378614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.378582 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\" (UniqueName: \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.378954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.378662 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.378954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.378748 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.378954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.378909 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.379175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379042 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.379175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379133 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\") pod \"488df13a-d848-449b-9d3a-6c275613ee4e\" (UID: \"488df13a-d848-449b-9d3a-6c275613ee4e\") " Oct 30 20:14:31.379688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379590 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:31.379869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379816 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:31.380417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379935 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.380417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.379954 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.380417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.380033 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:31.382108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.382081 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:31.382209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.382113 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "kube-api-access-rr6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:31.382391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.382377 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm" pod "488df13a-d848-449b-9d3a-6c275613ee4e" (UID: "488df13a-d848-449b-9d3a-6c275613ee4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:31.479760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.479729 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 20:14:31.480018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.479984 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" containerID="cri-o://5aacc19eaa3fa11a24274e5ecdd9d9a28763a4c20e078781f2e5a790523b9923" gracePeriod=30 Oct 30 20:14:31.480982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.480960 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr6wp\" (UniqueName: \"kubernetes.io/projected/488df13a-d848-449b-9d3a-6c275613ee4e-kube-api-access-rr6wp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.481078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.480987 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488df13a-d848-449b-9d3a-6c275613ee4e-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.481078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.481000 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.481078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.481011 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488df13a-d848-449b-9d3a-6c275613ee4e-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:31.603189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.603104 4959 generic.go:356] "Generic (PLEG): container finished" podID="488df13a-d848-449b-9d3a-6c275613ee4e" containerID="b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2" exitCode=0 Oct 30 20:14:31.603189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.603156 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" Oct 30 20:14:31.603430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.603281 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" event={"ID":"488df13a-d848-449b-9d3a-6c275613ee4e","Type":"ContainerDied","Data":"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2"} Oct 30 20:14:31.603430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.603318 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" event={"ID":"488df13a-d848-449b-9d3a-6c275613ee4e","Type":"ContainerDied","Data":"4b0e595c237ceb6679b85bdb09a7c579e8ea7c88d09751078f30a85fa3c57d26"} Oct 30 20:14:31.603430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.603340 4959 scope.go:122] "RemoveContainer" containerID="b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2" Oct 30 20:14:31.618408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.618388 4959 scope.go:122] "RemoveContainer" containerID="b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2" Oct 30 20:14:31.618806 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:31.618779 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2\": container with ID starting with b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2 not found: ID does not exist" containerID="b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2" Oct 30 20:14:31.618900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.618818 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2"} err="failed to get container status \"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2\": rpc error: code = NotFound desc = could not find container \"b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2\": container with ID starting with b13aed4b83d0ca418aa4a8fb0b726738445c9604fe9784ac3cd1cbd3dbb243d2 not found: ID does not exist" Oct 30 20:14:31.655524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.655476 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:14:31.659124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.659094 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s"] Oct 30 20:14:31.672674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.672653 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.855882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:31.855800 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.000961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:32.000932 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" path="/var/lib/kubelet/pods/488df13a-d848-449b-9d3a-6c275613ee4e/volumes" Oct 30 20:14:32.157015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:32.156983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.231541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:32.217295 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4a31d3/test-model-4a31d3-kserve-c6959558c-c4d5s" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.162:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:14:32.264123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:32.264083 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:34.075800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:34.075764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:34.693900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:34.693849 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" containerID="cri-o://cee95ccf7aab3675b2719a47f0f2553000c4f6fb99b385a41d0b3312c2a8b054" gracePeriod=25 Oct 30 20:14:35.809949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:35.809912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:36.377314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.377283 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:14:36.386068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.386036 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:14:36.645907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.645873 4959 generic.go:356] "Generic (PLEG): container finished" podID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerID="5aacc19eaa3fa11a24274e5ecdd9d9a28763a4c20e078781f2e5a790523b9923" exitCode=0 Oct 30 20:14:36.646079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.645938 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerDied","Data":"5aacc19eaa3fa11a24274e5ecdd9d9a28763a4c20e078781f2e5a790523b9923"} Oct 30 20:14:36.646079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.646003 4959 scope.go:122] "RemoveContainer" containerID="a456e0a59c5d80f86d93626f4839293350ec3e6ec38a1317d1c4558349801c1f" Oct 30 20:14:36.762954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.762930 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 20:14:36.846558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846526 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846599 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846728 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846787 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846833 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846924 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846958 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.846992 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847021 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847070 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\") pod \"850ff4e7-92c1-435f-86a6-54416b38c49d\" (UID: \"850ff4e7-92c1-435f-86a6-54416b38c49d\") " Oct 30 20:14:36.847419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847116 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:36.847419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847215 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.847419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847335 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.847761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847145 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.847761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847735 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.847761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847757 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/850ff4e7-92c1-435f-86a6-54416b38c49d-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.848085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847772 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.848085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847785 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.848085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.847797 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.849578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.849553 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:36.850135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.850111 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "kube-api-access-bp4qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:36.850255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.850149 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:36.851245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.851224 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy" pod "850ff4e7-92c1-435f-86a6-54416b38c49d" (UID: "850ff4e7-92c1-435f-86a6-54416b38c49d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.949681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.949599 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp4qs\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-kube-api-access-bp4qs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.949681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.949638 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.949681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.949647 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:36.949681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:36.949656 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/850ff4e7-92c1-435f-86a6-54416b38c49d-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:37.098682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.098652 4959 scope.go:122] "RemoveContainer" containerID="5aacc19eaa3fa11a24274e5ecdd9d9a28763a4c20e078781f2e5a790523b9923" Oct 30 20:14:37.501656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.501625 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:37.652808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.652776 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" event={"ID":"850ff4e7-92c1-435f-86a6-54416b38c49d","Type":"ContainerDied","Data":"8bc3ce4c3357860dbda3605d42aa6adc9d6b36e529bd307088757169246ee96d"} Oct 30 20:14:37.652987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.652898 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb" Oct 30 20:14:37.742883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.742850 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 20:14:37.748473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.748444 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f349a1-openshift-default-659765dc7-68tpb"] Oct 30 20:14:37.999046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:37.999015 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" path="/var/lib/kubelet/pods/850ff4e7-92c1-435f-86a6-54416b38c49d/volumes" Oct 30 20:14:38.268316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:38.268229 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:39.387841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:39.387808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:41.193522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:41.193465 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.217859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.217826 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:14:42.250334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.250294 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:14:42.314226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.314191 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:14:42.315263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315236 4959 cpu_manager.go:414] "containerMap: removing container" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" Oct 30 20:14:42.315263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315263 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315296 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315304 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315378 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e99fd189-0734-45dc-ac8a-f7f526288126" containerName="payload-processing" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315388 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99fd189-0734-45dc-ac8a-f7f526288126" containerName="payload-processing" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315402 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315409 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315431 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315438 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315456 4959 cpu_manager.go:414] "containerMap: removing container" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315463 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315474 4959 cpu_manager.go:414] "containerMap: removing container" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" Oct 30 20:14:42.315478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315482 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315516 4959 cpu_manager.go:414] "containerMap: removing container" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315524 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315538 4959 cpu_manager.go:414] "containerMap: removing container" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315545 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315556 4959 cpu_manager.go:414] "containerMap: removing container" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" containerName="main" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315563 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" containerName="main" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315577 4959 cpu_manager.go:414] "containerMap: removing container" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315583 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315592 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" containerName="maas-api" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315599 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" containerName="maas-api" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315616 4959 cpu_manager.go:414] "containerMap: removing container" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315622 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315640 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315648 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315666 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315679 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315693 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315701 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315713 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315720 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315915 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb1a303-3f04-46c3-aa38-7ba67a642cfd" containerName="main" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315927 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c3ac53-79b3-4839-8679-7bea0f99d22a" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315937 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315946 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="850ff4e7-92c1-435f-86a6-54416b38c49d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315960 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="488df13a-d848-449b-9d3a-6c275613ee4e" containerName="main" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315972 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae28612-2090-4280-9139-63ef293ba821" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315983 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4f37f9-488a-4f2e-a27c-a96b9033d2ac" containerName="maas-api" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.315992 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="440c558d-c52a-45f1-964e-bc090bd0268c" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316002 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be64621-aa39-4baa-aad0-c669440ee371" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316014 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af5edd2-6de4-49c1-b711-48d353cab69d" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316021 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad91e702-e7fc-4865-9ae6-1e61b631b593" containerName="payload-pre-processing" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316030 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c07e0b-d292-4832-81c2-d5198affed5b" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316043 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca576ba-0a1a-457c-be3a-c3bbaac08f0c" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316056 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c04781-2619-4bc2-af4e-6c873bd49547" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316068 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99fd189-0734-45dc-ac8a-f7f526288126" containerName="payload-processing" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316076 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a6dc58-5919-49b9-9165-466126251f9a" containerName="istio-proxy" Oct 30 20:14:42.316156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.316088 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a80b343-468d-46fe-b798-9bb1bcf76fd1" containerName="istio-proxy" Oct 30 20:14:42.326315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.326292 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.346198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.345586 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:14:42.346198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.345842 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a64be2-openshift-default-dockercfg-zbjbc\"" Oct 30 20:14:42.367524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.359450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.426359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426329 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426369 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426539 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xh5s\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426589 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426721 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426811 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426899 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.426974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426964 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.427014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.426999 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.495005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.494886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.528428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528395 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528488 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528579 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528638 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528695 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528742 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528847 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.528948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528886 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.528950 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xh5s\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.529068 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.529238 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.529255 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.529354 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.529778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.529752 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.531650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.531598 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.531788 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.531633 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.548567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.547321 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.548567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.547656 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xh5s\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\") pod \"e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.663888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.663860 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:42.968486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.968449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.989427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.989400 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:14:42.993560 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:14:42.993528 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20477bc9_8bb0_451f_be9a_0d618d4cfb59.slice/crio-c02af66ffa99639a06998558050fec19872756dc56230d29b93bc7baa4e1af89 WatchSource:0}: Error finding container c02af66ffa99639a06998558050fec19872756dc56230d29b93bc7baa4e1af89: Status 404 returned error can't find the container with id c02af66ffa99639a06998558050fec19872756dc56230d29b93bc7baa4e1af89 Oct 30 20:14:42.996609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.996574 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:14:42.996716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.996644 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:14:42.996716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:42.996692 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:14:43.698655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:43.698616 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" event={"ID":"20477bc9-8bb0-451f-be9a-0d618d4cfb59","Type":"ContainerStarted","Data":"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a"} Oct 30 20:14:43.699012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:43.698676 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" event={"ID":"20477bc9-8bb0-451f-be9a-0d618d4cfb59","Type":"ContainerStarted","Data":"c02af66ffa99639a06998558050fec19872756dc56230d29b93bc7baa4e1af89"} Oct 30 20:14:43.761396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:43.761347 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podStartSLOduration=1.761334867 podStartE2EDuration="1.761334867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:43.749786688 +0000 UTC m=+2407.879020609" watchObservedRunningTime="2026-10-30 20:14:43.761334867 +0000 UTC m=+2407.890568788" Oct 30 20:14:44.667520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:44.667326 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:44.667763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:44.667524 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:44.667763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:44.667575 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:44.704981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:44.704951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:45.669521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:45.666189 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:45.669521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:45.666250 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:46.091425 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:14:46.091338 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:14:46.414634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:46.414601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:46.665957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:46.665862 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:46.666231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:46.666198 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:47.218419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:47.218378 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" containerID="cri-o://9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b" gracePeriod=25 Oct 30 20:14:47.262610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:47.262566 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" containerID="cri-o://cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5" gracePeriod=25 Oct 30 20:14:47.666864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:47.666828 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:47.667144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:47.667115 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:48.099224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:48.099127 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.275807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:48.275752 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:48.665519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:48.664840 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:48.665519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:48.664910 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:49.432122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.432006 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:14:49.432713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.432231 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" podUID="6c62448d-e119-4af6-904e-0660b3bb00de" containerName="maas-api" containerID="cri-o://dcceb0fc9b37a572108541821ddaaca97510027e26aadcdff3f12ae94f758b9b" gracePeriod=30 Oct 30 20:14:49.492610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.492576 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.682324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.664986 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:14:49.682628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.682596 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 20:14:49.776055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.775874 4959 generic.go:356] "Generic (PLEG): container finished" podID="6c62448d-e119-4af6-904e-0660b3bb00de" containerID="dcceb0fc9b37a572108541821ddaaca97510027e26aadcdff3f12ae94f758b9b" exitCode=0 Oct 30 20:14:49.776055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.775972 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" event={"ID":"6c62448d-e119-4af6-904e-0660b3bb00de","Type":"ContainerDied","Data":"dcceb0fc9b37a572108541821ddaaca97510027e26aadcdff3f12ae94f758b9b"} Oct 30 20:14:49.820618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.820588 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:14:49.906693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.890599 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\") pod \"6c62448d-e119-4af6-904e-0660b3bb00de\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " Oct 30 20:14:49.906693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.890731 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\") pod \"6c62448d-e119-4af6-904e-0660b3bb00de\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " Oct 30 20:14:49.906693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.890934 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\" (UniqueName: \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\") pod \"6c62448d-e119-4af6-904e-0660b3bb00de\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " Oct 30 20:14:49.906693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.890977 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\") pod \"6c62448d-e119-4af6-904e-0660b3bb00de\" (UID: \"6c62448d-e119-4af6-904e-0660b3bb00de\") " Oct 30 20:14:49.928159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.928089 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs" pod "6c62448d-e119-4af6-904e-0660b3bb00de" (UID: "6c62448d-e119-4af6-904e-0660b3bb00de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:49.937549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.937444 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca" pod "6c62448d-e119-4af6-904e-0660b3bb00de" (UID: "6c62448d-e119-4af6-904e-0660b3bb00de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:49.951634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.947366 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz" pod "6c62448d-e119-4af6-904e-0660b3bb00de" (UID: "6c62448d-e119-4af6-904e-0660b3bb00de"). InnerVolumeSpecName "kube-api-access-b46xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:49.951634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.951591 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls" pod "6c62448d-e119-4af6-904e-0660b3bb00de" (UID: "6c62448d-e119-4af6-904e-0660b3bb00de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:49.993087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.993058 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b46xz\" (UniqueName: \"kubernetes.io/projected/6c62448d-e119-4af6-904e-0660b3bb00de-kube-api-access-b46xz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:49.993087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.993086 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:49.993087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.993095 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c62448d-e119-4af6-904e-0660b3bb00de-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:49.993396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:49.993104 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c62448d-e119-4af6-904e-0660b3bb00de-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:50.306867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.306785 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.418431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.418400 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.532227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.532193 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.670021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.669989 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:50.670524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.670508 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:50.671203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.671187 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:14:50.787151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.787107 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" event={"ID":"6c62448d-e119-4af6-904e-0660b3bb00de","Type":"ContainerDied","Data":"76a9d1868ee362578a43c49161c7c4c3d73ce8b9924a10d0a5ad736258309228"} Oct 30 20:14:50.787151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.787135 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk" Oct 30 20:14:50.787410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.787183 4959 scope.go:122] "RemoveContainer" containerID="dcceb0fc9b37a572108541821ddaaca97510027e26aadcdff3f12ae94f758b9b" Oct 30 20:14:50.843048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.842982 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:14:50.849087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.849058 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7b5635-cff4dd754-g9krk"] Oct 30 20:14:50.958241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:50.958166 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.001129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:52.001099 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c62448d-e119-4af6-904e-0660b3bb00de" path="/var/lib/kubelet/pods/6c62448d-e119-4af6-904e-0660b3bb00de/volumes" Oct 30 20:14:52.510972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:52.510936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.120465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:53.120432 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:54.128070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:54.128036 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:54.372244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:54.372210 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.820929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:54.820903 4959 generic.go:356] "Generic (PLEG): container finished" podID="f556501d-9a01-405a-8fde-af056b3af550" containerID="cee95ccf7aab3675b2719a47f0f2553000c4f6fb99b385a41d0b3312c2a8b054" exitCode=0 Oct 30 20:14:54.821074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:54.820979 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" event={"ID":"f556501d-9a01-405a-8fde-af056b3af550","Type":"ContainerDied","Data":"cee95ccf7aab3675b2719a47f0f2553000c4f6fb99b385a41d0b3312c2a8b054"} Oct 30 20:14:54.983432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:54.983410 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:14:55.058116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.058089 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\") pod \"f556501d-9a01-405a-8fde-af056b3af550\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " Oct 30 20:14:55.058274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.058200 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\" (UniqueName: \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\") pod \"f556501d-9a01-405a-8fde-af056b3af550\" (UID: \"f556501d-9a01-405a-8fde-af056b3af550\") " Oct 30 20:14:55.058562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.058534 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume" pod "f556501d-9a01-405a-8fde-af056b3af550" (UID: "f556501d-9a01-405a-8fde-af056b3af550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:55.059112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.058801 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f556501d-9a01-405a-8fde-af056b3af550-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:55.061536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.060890 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r" pod "f556501d-9a01-405a-8fde-af056b3af550" (UID: "f556501d-9a01-405a-8fde-af056b3af550"). InnerVolumeSpecName "kube-api-access-5ng8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:55.160084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.160050 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ng8r\" (UniqueName: \"kubernetes.io/projected/f556501d-9a01-405a-8fde-af056b3af550-kube-api-access-5ng8r\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:14:55.831062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.831037 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" Oct 30 20:14:55.831239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.831033 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn" event={"ID":"f556501d-9a01-405a-8fde-af056b3af550","Type":"ContainerDied","Data":"a244f5dda40a583fd534c83dbd5d353adc77479a27c003b3ba905ba2ce185b7d"} Oct 30 20:14:55.831239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.831169 4959 scope.go:122] "RemoveContainer" containerID="cee95ccf7aab3675b2719a47f0f2553000c4f6fb99b385a41d0b3312c2a8b054" Oct 30 20:14:55.881031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.881004 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:14:55.891761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.891734 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7b5635-54bcfbcdfb-xqqgn"] Oct 30 20:14:55.998851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:55.998822 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f556501d-9a01-405a-8fde-af056b3af550" path="/var/lib/kubelet/pods/f556501d-9a01-405a-8fde-af056b3af550/volumes" Oct 30 20:14:56.097858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:56.097791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.417353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:57.417317 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.525318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:57.525281 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.759244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:57.759162 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.870815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:57.870731 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:59.405255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:14:59.405221 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.193586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.193557 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:00.194441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194418 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" Oct 30 20:15:00.194441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194439 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" Oct 30 20:15:00.194639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194468 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6c62448d-e119-4af6-904e-0660b3bb00de" containerName="maas-api" Oct 30 20:15:00.194639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194474 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c62448d-e119-4af6-904e-0660b3bb00de" containerName="maas-api" Oct 30 20:15:00.194745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194661 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f556501d-9a01-405a-8fde-af056b3af550" containerName="payload-processing" Oct 30 20:15:00.194745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.194678 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c62448d-e119-4af6-904e-0660b3bb00de" containerName="maas-api" Oct 30 20:15:00.204814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.203338 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:00.204814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.204172 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.212277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.212242 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:00.212449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.212414 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.222749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.222726 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.239245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.239219 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:00.244789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.244768 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:15:00.262878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.262849 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:00.268604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.268580 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:00.272856 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.272834 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:00.281207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.281188 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.313776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.313744 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:00.323407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.321049 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:00.333205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.333181 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.343112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.342175 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:00.350759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.350725 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrmq\" (UniqueName: \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.350898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.350835 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrb2\" (UniqueName: \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.350993 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.350960 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.351203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.351049 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pbc\" (UniqueName: \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.351290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.351250 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.351348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.351308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.351420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.351401 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.351529 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.351512 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fgr\" (UniqueName: \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.362426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.362395 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:00.362582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.362541 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.380876 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.380853 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:00.434229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.434197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.453423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453351 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.453423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453413 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pbc\" (UniqueName: \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.453640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453445 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.453640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453476 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.453640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453510 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.453640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453539 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.453640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453590 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.453922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453641 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fgr\" (UniqueName: \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.453922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453764 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrmq\" (UniqueName: \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.453922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453833 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.453922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.453911 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrb2\" (UniqueName: \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.454147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.454008 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4zh\" (UniqueName: \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.454244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.454221 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.454293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.454272 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.454332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.454272 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.454372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.454348 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.491247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.491220 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrmq\" (UniqueName: \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.491421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.491395 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrb2\" (UniqueName: \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.491550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.491532 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fgr\" (UniqueName: \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.492200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.492176 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pbc\" (UniqueName: \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\") pod \"maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.527761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.527738 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:00.538361 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.538335 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:00.549746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.549722 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:00.563588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.563555 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.564364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.564336 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4zh\" (UniqueName: \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.566600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.564807 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.566600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.566440 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.567590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.567566 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.568557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.568532 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.571544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.569942 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.584526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.584471 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4zh\" (UniqueName: \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.585217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.585189 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\") pod \"maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.605668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.603125 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:00.673955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.673913 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.684671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.684639 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:00.694821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.694008 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:00.856196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.856039 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:00.861432 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:15:00.861401 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb83428_6848_477e_a30f_e002ed82a582.slice/crio-dc44197fb48c240329e78402795b629148019270e228d1710b8bd96f270e9dd8 WatchSource:0}: Error finding container dc44197fb48c240329e78402795b629148019270e228d1710b8bd96f270e9dd8: Status 404 returned error can't find the container with id dc44197fb48c240329e78402795b629148019270e228d1710b8bd96f270e9dd8 Oct 30 20:15:00.872373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.872350 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:00.880023 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:15:00.879992 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fe9a83_17bb_4325_a1ed_b2c3537e04af.slice/crio-2af27972bef51a07ff8877a62daaef7d40265a4c0951f71587753440b21641dd WatchSource:0}: Error finding container 2af27972bef51a07ff8877a62daaef7d40265a4c0951f71587753440b21641dd: Status 404 returned error can't find the container with id 2af27972bef51a07ff8877a62daaef7d40265a4c0951f71587753440b21641dd Oct 30 20:15:00.900294 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.900262 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerStarted","Data":"2af27972bef51a07ff8877a62daaef7d40265a4c0951f71587753440b21641dd"} Oct 30 20:15:00.902160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.902133 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerStarted","Data":"dc44197fb48c240329e78402795b629148019270e228d1710b8bd96f270e9dd8"} Oct 30 20:15:00.922029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.921994 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:00.954926 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:00.954902 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:00.978954 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:15:00.978917 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a400f76_2181_4479_ac00_a2da0716333d.slice/crio-63ac3826601a96eb13eb9a3b77af7cbe805e211ce0fcac2b940c2a2830d33f2c WatchSource:0}: Error finding container 63ac3826601a96eb13eb9a3b77af7cbe805e211ce0fcac2b940c2a2830d33f2c: Status 404 returned error can't find the container with id 63ac3826601a96eb13eb9a3b77af7cbe805e211ce0fcac2b940c2a2830d33f2c Oct 30 20:15:01.059107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.059079 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:01.062665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.062642 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:01.066998 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:15:01.066973 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1d3ed7_96ce_4dc3_b8e9_03c0932c2054.slice/crio-bb698dee8adacc0b10e583a29c45feebd908a169c4fe21756848e07f1780f56d WatchSource:0}: Error finding container bb698dee8adacc0b10e583a29c45feebd908a169c4fe21756848e07f1780f56d: Status 404 returned error can't find the container with id bb698dee8adacc0b10e583a29c45feebd908a169c4fe21756848e07f1780f56d Oct 30 20:15:01.067200 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:15:01.067179 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce4252b_e340_49c1_953a_33e6abb94ac2.slice/crio-7d3d59a8bc761be10d1f19dfece4c005055515223405193fed41cb12bf320af0 WatchSource:0}: Error finding container 7d3d59a8bc761be10d1f19dfece4c005055515223405193fed41cb12bf320af0: Status 404 returned error can't find the container with id 7d3d59a8bc761be10d1f19dfece4c005055515223405193fed41cb12bf320af0 Oct 30 20:15:01.099754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.099732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:01.913063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.913026 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerStarted","Data":"6cd2913f1d8ca9e7f970e2181d9a7af8f1b7039cf0d0c459354a245ca217cea4"} Oct 30 20:15:01.915058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.915028 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerStarted","Data":"c7d7aaec0409970f47148e2f5052eaef2207bf698291bd468affa8e9770b036c"} Oct 30 20:15:01.915191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.915091 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerStarted","Data":"bb698dee8adacc0b10e583a29c45feebd908a169c4fe21756848e07f1780f56d"} Oct 30 20:15:01.917168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.917142 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerStarted","Data":"c5b157d23ec374f0e638b1437f1a5f591a1cf3893a0b121736b15b9a22825027"} Oct 30 20:15:01.917348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.917189 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerStarted","Data":"7d3d59a8bc761be10d1f19dfece4c005055515223405193fed41cb12bf320af0"} Oct 30 20:15:01.919231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.919203 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerStarted","Data":"af5d95ff5253a89cbc4e0f7413d79d033cd28805e0010daef7c15bc43e1161b9"} Oct 30 20:15:01.919358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.919263 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerStarted","Data":"601fe57bc4d0a7aef45736b43c72044441a46c3298959e29c3be20ae2da6831b"} Oct 30 20:15:01.921328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.921300 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerStarted","Data":"9f8ed7853501d9f7c16e1528853bc149ab22e0456a944077f83a1ce5cba8cbff"} Oct 30 20:15:01.921413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.921353 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerStarted","Data":"63ac3826601a96eb13eb9a3b77af7cbe805e211ce0fcac2b940c2a2830d33f2c"} Oct 30 20:15:01.923837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.923816 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerStarted","Data":"174195bfa10637df7909669496a135a41dc9dbabb770aa5c0dee6470c051422d"} Oct 30 20:15:01.961934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:01.961892 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" podStartSLOduration=1.961879665 podStartE2EDuration="1.961879665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.961827272 +0000 UTC m=+2426.091061195" watchObservedRunningTime="2026-10-30 20:15:01.961879665 +0000 UTC m=+2426.091113580" Oct 30 20:15:02.019791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.019743 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" podStartSLOduration=2.019729052 podStartE2EDuration="2.019729052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.015325636 +0000 UTC m=+2426.144559558" watchObservedRunningTime="2026-10-30 20:15:02.019729052 +0000 UTC m=+2426.148962973" Oct 30 20:15:02.055628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.055582 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" podStartSLOduration=2.055569732 podStartE2EDuration="2.055569732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.055504593 +0000 UTC m=+2426.184738508" watchObservedRunningTime="2026-10-30 20:15:02.055569732 +0000 UTC m=+2426.184803653" Oct 30 20:15:02.096598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.096549 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" podStartSLOduration=2.096535146 podStartE2EDuration="2.096535146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.09227409 +0000 UTC m=+2426.221508015" watchObservedRunningTime="2026-10-30 20:15:02.096535146 +0000 UTC m=+2426.225769067" Oct 30 20:15:02.129707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.129662 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" podStartSLOduration=2.129650613 podStartE2EDuration="2.129650613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.12830093 +0000 UTC m=+2426.257534851" watchObservedRunningTime="2026-10-30 20:15:02.129650613 +0000 UTC m=+2426.258884572" Oct 30 20:15:02.744982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.744949 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:02.932878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.932845 4959 generic.go:356] "Generic (PLEG): container finished" podID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerID="6cd2913f1d8ca9e7f970e2181d9a7af8f1b7039cf0d0c459354a245ca217cea4" exitCode=60 Oct 30 20:15:02.933320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.932932 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerDied","Data":"6cd2913f1d8ca9e7f970e2181d9a7af8f1b7039cf0d0c459354a245ca217cea4"} Oct 30 20:15:02.933320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.933234 4959 scope.go:122] "RemoveContainer" containerID="6cd2913f1d8ca9e7f970e2181d9a7af8f1b7039cf0d0c459354a245ca217cea4" Oct 30 20:15:02.934735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.934715 4959 generic.go:356] "Generic (PLEG): container finished" podID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerID="c7d7aaec0409970f47148e2f5052eaef2207bf698291bd468affa8e9770b036c" exitCode=60 Oct 30 20:15:02.934862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.934783 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerDied","Data":"c7d7aaec0409970f47148e2f5052eaef2207bf698291bd468affa8e9770b036c"} Oct 30 20:15:02.935056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.935039 4959 scope.go:122] "RemoveContainer" containerID="c7d7aaec0409970f47148e2f5052eaef2207bf698291bd468affa8e9770b036c" Oct 30 20:15:02.936667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.936565 4959 generic.go:356] "Generic (PLEG): container finished" podID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerID="c5b157d23ec374f0e638b1437f1a5f591a1cf3893a0b121736b15b9a22825027" exitCode=60 Oct 30 20:15:02.936746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.936676 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerDied","Data":"c5b157d23ec374f0e638b1437f1a5f591a1cf3893a0b121736b15b9a22825027"} Oct 30 20:15:02.936853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.936834 4959 scope.go:122] "RemoveContainer" containerID="c5b157d23ec374f0e638b1437f1a5f591a1cf3893a0b121736b15b9a22825027" Oct 30 20:15:02.940286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.940248 4959 generic.go:356] "Generic (PLEG): container finished" podID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerID="af5d95ff5253a89cbc4e0f7413d79d033cd28805e0010daef7c15bc43e1161b9" exitCode=60 Oct 30 20:15:02.940400 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.940343 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerDied","Data":"af5d95ff5253a89cbc4e0f7413d79d033cd28805e0010daef7c15bc43e1161b9"} Oct 30 20:15:02.940631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.940612 4959 scope.go:122] "RemoveContainer" containerID="af5d95ff5253a89cbc4e0f7413d79d033cd28805e0010daef7c15bc43e1161b9" Oct 30 20:15:02.941846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.941829 4959 generic.go:356] "Generic (PLEG): container finished" podID="6a400f76-2181-4479-ac00-a2da0716333d" containerID="9f8ed7853501d9f7c16e1528853bc149ab22e0456a944077f83a1ce5cba8cbff" exitCode=60 Oct 30 20:15:02.941930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.941907 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerDied","Data":"9f8ed7853501d9f7c16e1528853bc149ab22e0456a944077f83a1ce5cba8cbff"} Oct 30 20:15:02.942109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.942094 4959 scope.go:122] "RemoveContainer" containerID="9f8ed7853501d9f7c16e1528853bc149ab22e0456a944077f83a1ce5cba8cbff" Oct 30 20:15:02.943438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.943413 4959 generic.go:356] "Generic (PLEG): container finished" podID="4eb83428-6848-477e-a30f-e002ed82a582" containerID="174195bfa10637df7909669496a135a41dc9dbabb770aa5c0dee6470c051422d" exitCode=60 Oct 30 20:15:02.943549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.943477 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerDied","Data":"174195bfa10637df7909669496a135a41dc9dbabb770aa5c0dee6470c051422d"} Oct 30 20:15:02.943672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.943659 4959 scope.go:122] "RemoveContainer" containerID="174195bfa10637df7909669496a135a41dc9dbabb770aa5c0dee6470c051422d" Oct 30 20:15:02.972638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:02.972590 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" podStartSLOduration=2.9725773650000002 podStartE2EDuration="2.972577365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.166081072 +0000 UTC m=+2426.295314993" watchObservedRunningTime="2026-10-30 20:15:02.972577365 +0000 UTC m=+2427.101811285" Oct 30 20:15:03.121382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.121339 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:03.962465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.962433 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerStarted","Data":"9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67"} Oct 30 20:15:03.964756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.964732 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerStarted","Data":"3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6"} Oct 30 20:15:03.966603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.966579 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerStarted","Data":"4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9"} Oct 30 20:15:03.968417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.968396 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerStarted","Data":"269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9"} Oct 30 20:15:03.970207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.970181 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerStarted","Data":"499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f"} Oct 30 20:15:03.971903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:03.971881 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerStarted","Data":"875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c"} Oct 30 20:15:04.128223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.128185 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:04.623917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.623811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.981077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.981045 4959 generic.go:356] "Generic (PLEG): container finished" podID="6a400f76-2181-4479-ac00-a2da0716333d" containerID="499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f" exitCode=60 Oct 30 20:15:04.981551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.981122 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerDied","Data":"499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f"} Oct 30 20:15:04.981551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.981183 4959 scope.go:122] "RemoveContainer" containerID="9f8ed7853501d9f7c16e1528853bc149ab22e0456a944077f83a1ce5cba8cbff" Oct 30 20:15:04.981551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.981466 4959 scope.go:122] "RemoveContainer" containerID="499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f" Oct 30 20:15:04.981881 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.981849 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz_odh-ai-gateway-infra(6a400f76-2181-4479-ac00-a2da0716333d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" podUID="6a400f76-2181-4479-ac00-a2da0716333d" Oct 30 20:15:04.983380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.983351 4959 generic.go:356] "Generic (PLEG): container finished" podID="4eb83428-6848-477e-a30f-e002ed82a582" containerID="875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c" exitCode=60 Oct 30 20:15:04.983474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.983430 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerDied","Data":"875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c"} Oct 30 20:15:04.983734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.983720 4959 scope.go:122] "RemoveContainer" containerID="875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c" Oct 30 20:15:04.984043 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.984021 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq_odh-ai-gateway-infra(4eb83428-6848-477e-a30f-e002ed82a582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" podUID="4eb83428-6848-477e-a30f-e002ed82a582" Oct 30 20:15:04.985796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.985762 4959 generic.go:356] "Generic (PLEG): container finished" podID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerID="9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67" exitCode=60 Oct 30 20:15:04.986091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.985790 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerDied","Data":"9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67"} Oct 30 20:15:04.986288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.986271 4959 scope.go:122] "RemoveContainer" containerID="9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67" Oct 30 20:15:04.986635 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.986613 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh_odh-ai-gateway-infra(35fe9a83-17bb-4325-a1ed-b2c3537e04af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" Oct 30 20:15:04.988241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.988217 4959 generic.go:356] "Generic (PLEG): container finished" podID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerID="3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6" exitCode=60 Oct 30 20:15:04.988329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.988254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerDied","Data":"3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6"} Oct 30 20:15:04.988532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.988515 4959 scope.go:122] "RemoveContainer" containerID="3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6" Oct 30 20:15:04.988879 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.988823 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l_odh-ai-gateway-infra(2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" Oct 30 20:15:04.990596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.990576 4959 generic.go:356] "Generic (PLEG): container finished" podID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerID="4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9" exitCode=60 Oct 30 20:15:04.990853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.990716 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerDied","Data":"4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9"} Oct 30 20:15:04.991124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.991098 4959 scope.go:122] "RemoveContainer" containerID="4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9" Oct 30 20:15:04.991435 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.991413 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct_odh-ai-gateway-infra(bce4252b-e340-49c1-953a-33e6abb94ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" Oct 30 20:15:04.992889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.992866 4959 generic.go:356] "Generic (PLEG): container finished" podID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerID="269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9" exitCode=60 Oct 30 20:15:04.992997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.992902 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerDied","Data":"269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9"} Oct 30 20:15:04.993131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.993117 4959 scope.go:122] "RemoveContainer" containerID="269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9" Oct 30 20:15:04.993354 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:04.993332 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98_odh-ai-gateway-infra(42ced1f3-94be-4cf7-ad4d-0014ce2cfde3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" Oct 30 20:15:04.996604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:04.996584 4959 scope.go:122] "RemoveContainer" containerID="174195bfa10637df7909669496a135a41dc9dbabb770aa5c0dee6470c051422d" Oct 30 20:15:05.010341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:05.010324 4959 scope.go:122] "RemoveContainer" containerID="6cd2913f1d8ca9e7f970e2181d9a7af8f1b7039cf0d0c459354a245ca217cea4" Oct 30 20:15:05.025630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:05.025608 4959 scope.go:122] "RemoveContainer" containerID="c7d7aaec0409970f47148e2f5052eaef2207bf698291bd468affa8e9770b036c" Oct 30 20:15:05.039560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:05.039538 4959 scope.go:122] "RemoveContainer" containerID="c5b157d23ec374f0e638b1437f1a5f591a1cf3893a0b121736b15b9a22825027" Oct 30 20:15:05.052797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:05.052776 4959 scope.go:122] "RemoveContainer" containerID="af5d95ff5253a89cbc4e0f7413d79d033cd28805e0010daef7c15bc43e1161b9" Oct 30 20:15:06.016305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.016271 4959 scope.go:122] "RemoveContainer" containerID="269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9" Oct 30 20:15:06.016742 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.016574 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98_odh-ai-gateway-infra(42ced1f3-94be-4cf7-ad4d-0014ce2cfde3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" Oct 30 20:15:06.018076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.018058 4959 scope.go:122] "RemoveContainer" containerID="499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f" Oct 30 20:15:06.018289 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.018267 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz_odh-ai-gateway-infra(6a400f76-2181-4479-ac00-a2da0716333d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" podUID="6a400f76-2181-4479-ac00-a2da0716333d" Oct 30 20:15:06.019577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.019557 4959 scope.go:122] "RemoveContainer" containerID="875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c" Oct 30 20:15:06.019815 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.019793 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq_odh-ai-gateway-infra(4eb83428-6848-477e-a30f-e002ed82a582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" podUID="4eb83428-6848-477e-a30f-e002ed82a582" Oct 30 20:15:06.021088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.021070 4959 scope.go:122] "RemoveContainer" containerID="9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67" Oct 30 20:15:06.021301 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.021282 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh_odh-ai-gateway-infra(35fe9a83-17bb-4325-a1ed-b2c3537e04af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" Oct 30 20:15:06.022617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.022599 4959 scope.go:122] "RemoveContainer" containerID="3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6" Oct 30 20:15:06.022815 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.022793 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l_odh-ai-gateway-infra(2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" Oct 30 20:15:06.024108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.024090 4959 scope.go:122] "RemoveContainer" containerID="4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9" Oct 30 20:15:06.024317 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:06.024297 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct_odh-ai-gateway-infra(bce4252b-e340-49c1-953a-33e6abb94ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" Oct 30 20:15:06.174260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:06.174226 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.057725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.057691 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.181969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.181931 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.307664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.307639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.601245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.601220 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:15:07.625540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.625518 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:15:07.689642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.689565 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\" (UniqueName: \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\") pod \"740a4953-6deb-48e4-90cd-b5d09c1b9706\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " Oct 30 20:15:07.689790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.689764 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\") pod \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " Oct 30 20:15:07.690210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.690178 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume" pod "d7c3d7f6-14af-433c-9db2-553060cfa7cc" (UID: "d7c3d7f6-14af-433c-9db2-553060cfa7cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:07.690479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.690287 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\") pod \"740a4953-6deb-48e4-90cd-b5d09c1b9706\" (UID: \"740a4953-6deb-48e4-90cd-b5d09c1b9706\") " Oct 30 20:15:07.690479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.690373 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\" (UniqueName: \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\") pod \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\" (UID: \"d7c3d7f6-14af-433c-9db2-553060cfa7cc\") " Oct 30 20:15:07.690673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.690647 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume" pod "740a4953-6deb-48e4-90cd-b5d09c1b9706" (UID: "740a4953-6deb-48e4-90cd-b5d09c1b9706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:07.691205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.691180 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740a4953-6deb-48e4-90cd-b5d09c1b9706-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:07.691300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.691212 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c3d7f6-14af-433c-9db2-553060cfa7cc-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:07.692580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.692555 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws" pod "740a4953-6deb-48e4-90cd-b5d09c1b9706" (UID: "740a4953-6deb-48e4-90cd-b5d09c1b9706"). InnerVolumeSpecName "kube-api-access-wlvws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:07.693312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.693284 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls" pod "d7c3d7f6-14af-433c-9db2-553060cfa7cc" (UID: "d7c3d7f6-14af-433c-9db2-553060cfa7cc"). InnerVolumeSpecName "kube-api-access-bvpls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:07.738069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.738041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.792314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.792285 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvpls\" (UniqueName: \"kubernetes.io/projected/d7c3d7f6-14af-433c-9db2-553060cfa7cc-kube-api-access-bvpls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:07.792314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:07.792313 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvws\" (UniqueName: \"kubernetes.io/projected/740a4953-6deb-48e4-90cd-b5d09c1b9706-kube-api-access-wlvws\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:08.039721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.039689 4959 generic.go:356] "Generic (PLEG): container finished" podID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerID="cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5" exitCode=0 Oct 30 20:15:08.039854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.039737 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" Oct 30 20:15:08.039854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.039768 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" event={"ID":"d7c3d7f6-14af-433c-9db2-553060cfa7cc","Type":"ContainerDied","Data":"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5"} Oct 30 20:15:08.039854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.039821 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4" event={"ID":"d7c3d7f6-14af-433c-9db2-553060cfa7cc","Type":"ContainerDied","Data":"42c2242d347bb829d0b8fae7111ecaca40f40a18ab25a55b37e7c602a0c34ad9"} Oct 30 20:15:08.039854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.039851 4959 scope.go:122] "RemoveContainer" containerID="cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5" Oct 30 20:15:08.041427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.041403 4959 generic.go:356] "Generic (PLEG): container finished" podID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerID="9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b" exitCode=0 Oct 30 20:15:08.041562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.041442 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" Oct 30 20:15:08.041562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.041469 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" event={"ID":"740a4953-6deb-48e4-90cd-b5d09c1b9706","Type":"ContainerDied","Data":"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b"} Oct 30 20:15:08.041562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.041524 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q" event={"ID":"740a4953-6deb-48e4-90cd-b5d09c1b9706","Type":"ContainerDied","Data":"ba1486b74c97861e0a6b25bd5a0db3d51bba4dd002559ff128262fc1d497397b"} Oct 30 20:15:08.050720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.050704 4959 scope.go:122] "RemoveContainer" containerID="cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5" Oct 30 20:15:08.050969 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:08.050951 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5\": container with ID starting with cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5 not found: ID does not exist" containerID="cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5" Oct 30 20:15:08.051048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.050978 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5"} err="failed to get container status \"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5\": rpc error: code = NotFound desc = could not find container \"cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5\": container with ID starting with cf53fb42f8e206e37c3cd6511154c5f729eb77bf716222fb604ed09b5b2ae7f5 not found: ID does not exist" Oct 30 20:15:08.051048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.050999 4959 scope.go:122] "RemoveContainer" containerID="9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b" Oct 30 20:15:08.061894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.061877 4959 scope.go:122] "RemoveContainer" containerID="9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b" Oct 30 20:15:08.062183 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:08.062162 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b\": container with ID starting with 9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b not found: ID does not exist" containerID="9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b" Oct 30 20:15:08.062231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.062181 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b"} err="failed to get container status \"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b\": rpc error: code = NotFound desc = could not find container \"9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b\": container with ID starting with 9015a2468de8b89b80c7b87ad5d7d06e38c6bf57ad8b452e7e97fbe68e93d13b not found: ID does not exist" Oct 30 20:15:08.089786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.089765 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:15:08.100359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.100337 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c17154-55ff656d7d-2j85q"] Oct 30 20:15:08.125557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.125537 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:15:08.133912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.133890 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17154-57bf6fbc9f-d69m4"] Oct 30 20:15:08.487649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.487619 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:15:08.521654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.521621 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:15:08.543478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.543456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.657115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.657089 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.772948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.772856 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.903882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:08.903849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:09.311785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:09.311748 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:09.999600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:09.999563 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" path="/var/lib/kubelet/pods/740a4953-6deb-48e4-90cd-b5d09c1b9706/volumes" Oct 30 20:15:10.000135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:10.000119 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" path="/var/lib/kubelet/pods/d7c3d7f6-14af-433c-9db2-553060cfa7cc/volumes" Oct 30 20:15:11.299478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:11.299447 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.102431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:13.102377 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.488059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:13.488016 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" containerID="cri-o://4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697" gracePeriod=25 Oct 30 20:15:13.522859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:13.522821 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" containerID="cri-o://d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a" gracePeriod=25 Oct 30 20:15:13.801580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:13.801474 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.912335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:13.912304 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.150463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:14.150409 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.989578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:14.989548 4959 scope.go:122] "RemoveContainer" containerID="9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67" Oct 30 20:15:15.212246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:15.212217 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.660300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:15.660263 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:15.660455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:15.660360 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:16.000731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.000647 4959 scope.go:122] "RemoveContainer" containerID="269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9" Oct 30 20:15:16.066814 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:16.066780 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:15:16.105006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.104982 4959 generic.go:356] "Generic (PLEG): container finished" podID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerID="637107b15dd761f97eb57e02c21c1597db40850d280a5b76893060734065caf8" exitCode=60 Oct 30 20:15:16.105138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.105058 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerDied","Data":"637107b15dd761f97eb57e02c21c1597db40850d280a5b76893060734065caf8"} Oct 30 20:15:16.105138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.105116 4959 scope.go:122] "RemoveContainer" containerID="9b8ab4427a387f25de6908f1d2234bcaa2ad70670f038c531d9810229acf2d67" Oct 30 20:15:16.105470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.105451 4959 scope.go:122] "RemoveContainer" containerID="637107b15dd761f97eb57e02c21c1597db40850d280a5b76893060734065caf8" Oct 30 20:15:16.105856 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:16.105837 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh_odh-ai-gateway-infra(35fe9a83-17bb-4325-a1ed-b2c3537e04af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" Oct 30 20:15:16.989110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:16.989083 4959 scope.go:122] "RemoveContainer" containerID="3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6" Oct 30 20:15:17.058010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.057986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:17.132864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.132837 4959 generic.go:356] "Generic (PLEG): container finished" podID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerID="b2b4c887f8978e84bdde32be1206c5e24e03c0314abd800cee5eb140ebb0d39b" exitCode=60 Oct 30 20:15:17.133045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.132914 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerDied","Data":"b2b4c887f8978e84bdde32be1206c5e24e03c0314abd800cee5eb140ebb0d39b"} Oct 30 20:15:17.133045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.132985 4959 scope.go:122] "RemoveContainer" containerID="269fd089170a6c7ca7ff4b096c954b79d509a4bd41b7d5a9d47a833b67da66f9" Oct 30 20:15:17.133440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.133422 4959 scope.go:122] "RemoveContainer" containerID="b2b4c887f8978e84bdde32be1206c5e24e03c0314abd800cee5eb140ebb0d39b" Oct 30 20:15:17.133843 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:17.133822 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98_odh-ai-gateway-infra(42ced1f3-94be-4cf7-ad4d-0014ce2cfde3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" Oct 30 20:15:17.164807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.164784 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:17.383793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.383741 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:17.549255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.549224 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\" (UniqueName: \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\") pod \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " Oct 30 20:15:17.549451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.549400 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\") pod \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\" (UID: \"35fe9a83-17bb-4325-a1ed-b2c3537e04af\") " Oct 30 20:15:17.549882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.549853 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca" pod "35fe9a83-17bb-4325-a1ed-b2c3537e04af" (UID: "35fe9a83-17bb-4325-a1ed-b2c3537e04af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:17.550055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.550038 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35fe9a83-17bb-4325-a1ed-b2c3537e04af-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:17.551404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.551378 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr" pod "35fe9a83-17bb-4325-a1ed-b2c3537e04af" (UID: "35fe9a83-17bb-4325-a1ed-b2c3537e04af"). InnerVolumeSpecName "kube-api-access-d7fgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:17.651798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.651732 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fgr\" (UniqueName: \"kubernetes.io/projected/35fe9a83-17bb-4325-a1ed-b2c3537e04af-kube-api-access-d7fgr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:17.989237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.989208 4959 scope.go:122] "RemoveContainer" containerID="499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f" Oct 30 20:15:17.989654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.989270 4959 scope.go:122] "RemoveContainer" containerID="875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c" Oct 30 20:15:17.989654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:17.989350 4959 scope.go:122] "RemoveContainer" containerID="4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9" Oct 30 20:15:18.141295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.141268 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" Oct 30 20:15:18.141295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.141284 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh" event={"ID":"35fe9a83-17bb-4325-a1ed-b2c3537e04af","Type":"ContainerDied","Data":"2af27972bef51a07ff8877a62daaef7d40265a4c0951f71587753440b21641dd"} Oct 30 20:15:18.141558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.141340 4959 scope.go:122] "RemoveContainer" containerID="637107b15dd761f97eb57e02c21c1597db40850d280a5b76893060734065caf8" Oct 30 20:15:18.143594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.143570 4959 generic.go:356] "Generic (PLEG): container finished" podID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerID="7c9abc615c9747f29103429ce773e3844d916ccab5f2f22a916b57e2ae23da5e" exitCode=60 Oct 30 20:15:18.143715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.143631 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerDied","Data":"7c9abc615c9747f29103429ce773e3844d916ccab5f2f22a916b57e2ae23da5e"} Oct 30 20:15:18.144066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.144051 4959 scope.go:122] "RemoveContainer" containerID="7c9abc615c9747f29103429ce773e3844d916ccab5f2f22a916b57e2ae23da5e" Oct 30 20:15:18.144309 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:18.144291 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l_odh-ai-gateway-infra(2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" Oct 30 20:15:18.155347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.155329 4959 scope.go:122] "RemoveContainer" containerID="3619392afe656d5ebce4a41c1cec6967cd26c4be3b3fdd6d4b6f2684d7a939c6" Oct 30 20:15:18.184601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.184574 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:18.232104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.232028 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:18.235653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.235624 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-284bf6-29889855-zmjhh"] Oct 30 20:15:18.485799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.485751 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:18.666007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.665979 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\" (UniqueName: \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\") pod \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " Oct 30 20:15:18.666172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.666042 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\") pod \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\" (UID: \"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3\") " Oct 30 20:15:18.666419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.666388 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca" pod "42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" (UID: "42ced1f3-94be-4cf7-ad4d-0014ce2cfde3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:18.666753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.666697 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.668320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.668290 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq" pod "42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" (UID: "42ced1f3-94be-4cf7-ad4d-0014ce2cfde3"). InnerVolumeSpecName "kube-api-access-shrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:18.767963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.767894 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shrmq\" (UniqueName: \"kubernetes.io/projected/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3-kube-api-access-shrmq\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:18.987074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:18.987039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.163794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.163761 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerStarted","Data":"5b0498f418485d7aeba5d96aafd68960628c3a155e5959ef5b65aae305dc93c6"} Oct 30 20:15:19.165069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.165042 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" Oct 30 20:15:19.165176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.165068 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98" event={"ID":"42ced1f3-94be-4cf7-ad4d-0014ce2cfde3","Type":"ContainerDied","Data":"601fe57bc4d0a7aef45736b43c72044441a46c3298959e29c3be20ae2da6831b"} Oct 30 20:15:19.165176 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.165119 4959 scope.go:122] "RemoveContainer" containerID="b2b4c887f8978e84bdde32be1206c5e24e03c0314abd800cee5eb140ebb0d39b" Oct 30 20:15:19.167304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.167278 4959 generic.go:356] "Generic (PLEG): container finished" podID="6a400f76-2181-4479-ac00-a2da0716333d" containerID="7fbdea7b56a91e5f781b48124135a2d11e4beb22b08e6a2b308c7214f8c1210b" exitCode=60 Oct 30 20:15:19.167429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.167350 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerDied","Data":"7fbdea7b56a91e5f781b48124135a2d11e4beb22b08e6a2b308c7214f8c1210b"} Oct 30 20:15:19.167859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.167837 4959 scope.go:122] "RemoveContainer" containerID="7fbdea7b56a91e5f781b48124135a2d11e4beb22b08e6a2b308c7214f8c1210b" Oct 30 20:15:19.168221 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:19.168185 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz_odh-ai-gateway-infra(6a400f76-2181-4479-ac00-a2da0716333d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" podUID="6a400f76-2181-4479-ac00-a2da0716333d" Oct 30 20:15:19.169540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.169515 4959 generic.go:356] "Generic (PLEG): container finished" podID="4eb83428-6848-477e-a30f-e002ed82a582" containerID="2b743e4db2a85f7e9dcabf8b6830930b0dcfd035e33732a293e227d4f46f7661" exitCode=60 Oct 30 20:15:19.169638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.169522 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerDied","Data":"2b743e4db2a85f7e9dcabf8b6830930b0dcfd035e33732a293e227d4f46f7661"} Oct 30 20:15:19.169958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.169945 4959 scope.go:122] "RemoveContainer" containerID="2b743e4db2a85f7e9dcabf8b6830930b0dcfd035e33732a293e227d4f46f7661" Oct 30 20:15:19.170211 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:19.170196 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq_odh-ai-gateway-infra(4eb83428-6848-477e-a30f-e002ed82a582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" podUID="4eb83428-6848-477e-a30f-e002ed82a582" Oct 30 20:15:19.177189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.177169 4959 scope.go:122] "RemoveContainer" containerID="499dfaf6d7eb6abed36e8a9ba91e6de51be7e82d80b77f31b2d81a2ad3df2d4f" Oct 30 20:15:19.190861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.190845 4959 scope.go:122] "RemoveContainer" containerID="875d609d5ee7c95598a2d5f10a80db5e1c0a11a122c92a01fddff3ce4548f36c" Oct 30 20:15:19.208956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.208936 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:19.238060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.238029 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:19.241352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.241328 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-043892-29889855-h7f98"] Oct 30 20:15:19.294259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.294217 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.381604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.381582 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:19.415200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.415175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.476651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.476618 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\") pod \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " Oct 30 20:15:19.476913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.476879 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\") pod \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\" (UID: \"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054\") " Oct 30 20:15:19.477314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.477282 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca" pod "2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" (UID: "2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.477608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.477585 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.478970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.478943 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2" pod "2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" (UID: "2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054"). InnerVolumeSpecName "kube-api-access-d2wh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.583650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.583577 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wh2\" (UniqueName: \"kubernetes.io/projected/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054-kube-api-access-d2wh2\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:19.700355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.700327 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:15:19.727675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.727645 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:15:19.847243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.847159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.958107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:19.958072 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.003145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.003079 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" path="/var/lib/kubelet/pods/35fe9a83-17bb-4325-a1ed-b2c3537e04af/volumes" Oct 30 20:15:20.008922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.008892 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" path="/var/lib/kubelet/pods/42ced1f3-94be-4cf7-ad4d-0014ce2cfde3/volumes" Oct 30 20:15:20.188565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.188534 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" event={"ID":"2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054","Type":"ContainerDied","Data":"bb698dee8adacc0b10e583a29c45feebd908a169c4fe21756848e07f1780f56d"} Oct 30 20:15:20.188565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.188556 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l" Oct 30 20:15:20.189134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.188600 4959 scope.go:122] "RemoveContainer" containerID="7c9abc615c9747f29103429ce773e3844d916ccab5f2f22a916b57e2ae23da5e" Oct 30 20:15:20.191978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.191950 4959 generic.go:356] "Generic (PLEG): container finished" podID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerID="5b0498f418485d7aeba5d96aafd68960628c3a155e5959ef5b65aae305dc93c6" exitCode=60 Oct 30 20:15:20.192101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.191996 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerDied","Data":"5b0498f418485d7aeba5d96aafd68960628c3a155e5959ef5b65aae305dc93c6"} Oct 30 20:15:20.192414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.192392 4959 scope.go:122] "RemoveContainer" containerID="5b0498f418485d7aeba5d96aafd68960628c3a155e5959ef5b65aae305dc93c6" Oct 30 20:15:20.192772 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:20.192737 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct_odh-ai-gateway-infra(bce4252b-e340-49c1-953a-33e6abb94ac2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" Oct 30 20:15:20.207416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.207399 4959 scope.go:122] "RemoveContainer" containerID="4b53bbedf42ea505631609e8c0410443c83d060960a612e883a9d3b43a7f10b9" Oct 30 20:15:20.211305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.211288 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:20.272384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.272354 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:20.277117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.277094 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a64be2-29889855-mqf7l"] Oct 30 20:15:20.287999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.287980 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:20.406615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.406097 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:20.572771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.572744 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:20.603297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.603258 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\") pod \"6a400f76-2181-4479-ac00-a2da0716333d\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " Oct 30 20:15:20.603429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.603365 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\" (UniqueName: \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\") pod \"6a400f76-2181-4479-ac00-a2da0716333d\" (UID: \"6a400f76-2181-4479-ac00-a2da0716333d\") " Oct 30 20:15:20.603612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.603589 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca" pod "6a400f76-2181-4479-ac00-a2da0716333d" (UID: "6a400f76-2181-4479-ac00-a2da0716333d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:20.604170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.604142 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a400f76-2181-4479-ac00-a2da0716333d-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:20.605862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.605836 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2" pod "6a400f76-2181-4479-ac00-a2da0716333d" (UID: "6a400f76-2181-4479-ac00-a2da0716333d"). InnerVolumeSpecName "kube-api-access-4xrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:20.640206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.640179 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:20.704771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.704677 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\") pod \"4eb83428-6848-477e-a30f-e002ed82a582\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " Oct 30 20:15:20.704771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.704751 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\" (UniqueName: \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\") pod \"4eb83428-6848-477e-a30f-e002ed82a582\" (UID: \"4eb83428-6848-477e-a30f-e002ed82a582\") " Oct 30 20:15:20.705408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.705373 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca" pod "4eb83428-6848-477e-a30f-e002ed82a582" (UID: "4eb83428-6848-477e-a30f-e002ed82a582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:20.705573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.705454 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrb2\" (UniqueName: \"kubernetes.io/projected/6a400f76-2181-4479-ac00-a2da0716333d-kube-api-access-4xrb2\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:20.707514 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.707472 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc" pod "4eb83428-6848-477e-a30f-e002ed82a582" (UID: "4eb83428-6848-477e-a30f-e002ed82a582"). InnerVolumeSpecName "kube-api-access-p9pbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:20.806428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.806399 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb83428-6848-477e-a30f-e002ed82a582-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:20.806428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.806424 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pbc\" (UniqueName: \"kubernetes.io/projected/4eb83428-6848-477e-a30f-e002ed82a582-kube-api-access-p9pbc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:20.960203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:20.960116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:21.207454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.207430 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" Oct 30 20:15:21.207454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.207442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz" event={"ID":"6a400f76-2181-4479-ac00-a2da0716333d","Type":"ContainerDied","Data":"63ac3826601a96eb13eb9a3b77af7cbe805e211ce0fcac2b940c2a2830d33f2c"} Oct 30 20:15:21.207946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.207523 4959 scope.go:122] "RemoveContainer" containerID="7fbdea7b56a91e5f781b48124135a2d11e4beb22b08e6a2b308c7214f8c1210b" Oct 30 20:15:21.209297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.209277 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" Oct 30 20:15:21.209410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.209367 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq" event={"ID":"4eb83428-6848-477e-a30f-e002ed82a582","Type":"ContainerDied","Data":"dc44197fb48c240329e78402795b629148019270e228d1710b8bd96f270e9dd8"} Oct 30 20:15:21.218507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.218463 4959 scope.go:122] "RemoveContainer" containerID="2b743e4db2a85f7e9dcabf8b6830930b0dcfd035e33732a293e227d4f46f7661" Oct 30 20:15:21.284213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.284180 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:21.288398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.288371 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889855-ztnwz"] Oct 30 20:15:21.319355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.319327 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:21.324776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.324749 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f349a1-29889855-8mqxq"] Oct 30 20:15:21.386766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.386735 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:21.414478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.414449 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\" (UniqueName: \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\") pod \"bce4252b-e340-49c1-953a-33e6abb94ac2\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " Oct 30 20:15:21.414872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.414846 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\") pod \"bce4252b-e340-49c1-953a-33e6abb94ac2\" (UID: \"bce4252b-e340-49c1-953a-33e6abb94ac2\") " Oct 30 20:15:21.415248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.415227 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca" pod "bce4252b-e340-49c1-953a-33e6abb94ac2" (UID: "bce4252b-e340-49c1-953a-33e6abb94ac2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.415480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.415447 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce4252b-e340-49c1-953a-33e6abb94ac2-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:21.416998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.416971 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh" pod "bce4252b-e340-49c1-953a-33e6abb94ac2" (UID: "bce4252b-e340-49c1-953a-33e6abb94ac2"). InnerVolumeSpecName "kube-api-access-gh4zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.516937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:21.516860 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh4zh\" (UniqueName: \"kubernetes.io/projected/bce4252b-e340-49c1-953a-33e6abb94ac2-kube-api-access-gh4zh\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:22.000128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.000089 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" path="/var/lib/kubelet/pods/2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054/volumes" Oct 30 20:15:22.000669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.000646 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb83428-6848-477e-a30f-e002ed82a582" path="/var/lib/kubelet/pods/4eb83428-6848-477e-a30f-e002ed82a582/volumes" Oct 30 20:15:22.001114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.001093 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a400f76-2181-4479-ac00-a2da0716333d" path="/var/lib/kubelet/pods/6a400f76-2181-4479-ac00-a2da0716333d/volumes" Oct 30 20:15:22.222563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.222518 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" Oct 30 20:15:22.222563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.222521 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct" event={"ID":"bce4252b-e340-49c1-953a-33e6abb94ac2","Type":"ContainerDied","Data":"7d3d59a8bc761be10d1f19dfece4c005055515223405193fed41cb12bf320af0"} Oct 30 20:15:22.223113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.222595 4959 scope.go:122] "RemoveContainer" containerID="5b0498f418485d7aeba5d96aafd68960628c3a155e5959ef5b65aae305dc93c6" Oct 30 20:15:22.275762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.275691 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:22.288345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.288319 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889855-zh5ct"] Oct 30 20:15:22.729342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:22.729292 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:24.001024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:24.000993 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" path="/var/lib/kubelet/pods/bce4252b-e340-49c1-953a-33e6abb94ac2/volumes" Oct 30 20:15:24.504144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:24.504106 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:24.700933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:24.700880 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" podUID="2ac7786e-fbd5-4d81-af38-1b336196bd05" containerName="payload-processing" containerID="cri-o://4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade" gracePeriod=25 Oct 30 20:15:24.728329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:24.728297 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" containerID="cri-o://47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211" gracePeriod=25 Oct 30 20:15:24.988384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:24.988352 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:15:25.059343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.059291 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\" (UniqueName: \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\") pod \"2ac7786e-fbd5-4d81-af38-1b336196bd05\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " Oct 30 20:15:25.059947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.059471 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\") pod \"2ac7786e-fbd5-4d81-af38-1b336196bd05\" (UID: \"2ac7786e-fbd5-4d81-af38-1b336196bd05\") " Oct 30 20:15:25.059947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.059925 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume" pod "2ac7786e-fbd5-4d81-af38-1b336196bd05" (UID: "2ac7786e-fbd5-4d81-af38-1b336196bd05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:25.060197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.060174 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac7786e-fbd5-4d81-af38-1b336196bd05-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:25.062982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.062953 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws" pod "2ac7786e-fbd5-4d81-af38-1b336196bd05" (UID: "2ac7786e-fbd5-4d81-af38-1b336196bd05"). InnerVolumeSpecName "kube-api-access-54cws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:25.161699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.161668 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54cws\" (UniqueName: \"kubernetes.io/projected/2ac7786e-fbd5-4d81-af38-1b336196bd05-kube-api-access-54cws\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:25.247534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.247486 4959 generic.go:356] "Generic (PLEG): container finished" podID="2ac7786e-fbd5-4d81-af38-1b336196bd05" containerID="4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade" exitCode=0 Oct 30 20:15:25.247710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.247532 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" event={"ID":"2ac7786e-fbd5-4d81-af38-1b336196bd05","Type":"ContainerDied","Data":"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade"} Oct 30 20:15:25.247710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.247550 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" Oct 30 20:15:25.247710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.247602 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr" event={"ID":"2ac7786e-fbd5-4d81-af38-1b336196bd05","Type":"ContainerDied","Data":"bbf8ca110dbfb0a2ef5b0e3c7271adde149c0bac3cdc7dd0017286a0e48c4010"} Oct 30 20:15:25.247710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.247633 4959 scope.go:122] "RemoveContainer" containerID="4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade" Oct 30 20:15:25.259901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.259883 4959 scope.go:122] "RemoveContainer" containerID="4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade" Oct 30 20:15:25.260184 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:25.260165 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade\": container with ID starting with 4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade not found: ID does not exist" containerID="4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade" Oct 30 20:15:25.260242 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.260196 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade"} err="failed to get container status \"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade\": rpc error: code = NotFound desc = could not find container \"4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade\": container with ID starting with 4a9d9c323252fb07f81a7d3fad318906f08bf74fbbf3cfaaa9cb1d9f7f04cade not found: ID does not exist" Oct 30 20:15:25.296566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.296544 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:15:25.307414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.307395 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6cd2ad-7b69dc4bb-64lfr"] Oct 30 20:15:25.466540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.466516 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:15:25.579200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.579159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.695434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.695394 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:25.696332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.696302 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:25.697832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.697703 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.999431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:25.999395 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac7786e-fbd5-4d81-af38-1b336196bd05" path="/var/lib/kubelet/pods/2ac7786e-fbd5-4d81-af38-1b336196bd05/volumes" Oct 30 20:15:26.256063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:26.255953 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:27.986471 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:27.986418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.061756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:30.061722 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.121750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:30.121630 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:30.467139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:30.467078 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" containerID="cri-o://07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e" gracePeriod=25 Oct 30 20:15:31.922378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:31.922312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.031102 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:32.031075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.362559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:32.362440 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.652541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.652489 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.886621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.886597 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:15:33.890122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.890101 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:15:33.983015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.982976 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\" (UniqueName: \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\") pod \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " Oct 30 20:15:33.983183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.983075 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\" (UniqueName: \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\") pod \"86a19396-fc1d-4247-bd55-b5e8f08d3623\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " Oct 30 20:15:33.983183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.983136 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\") pod \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\" (UID: \"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4\") " Oct 30 20:15:33.983317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.983302 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\") pod \"86a19396-fc1d-4247-bd55-b5e8f08d3623\" (UID: \"86a19396-fc1d-4247-bd55-b5e8f08d3623\") " Oct 30 20:15:33.983797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.983768 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume" pod "bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" (UID: "bc6260a9-9d30-47fc-9348-5d2ca42b3bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:33.984240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.984214 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume" pod "86a19396-fc1d-4247-bd55-b5e8f08d3623" (UID: "86a19396-fc1d-4247-bd55-b5e8f08d3623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:33.985623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.985599 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p" pod "86a19396-fc1d-4247-bd55-b5e8f08d3623" (UID: "86a19396-fc1d-4247-bd55-b5e8f08d3623"). InnerVolumeSpecName "kube-api-access-6cr8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:33.985720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:33.985642 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb" pod "bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" (UID: "bc6260a9-9d30-47fc-9348-5d2ca42b3bd4"). InnerVolumeSpecName "kube-api-access-l9vlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:34.085371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.085304 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vlb\" (UniqueName: \"kubernetes.io/projected/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-kube-api-access-l9vlb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:34.085371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.085332 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cr8p\" (UniqueName: \"kubernetes.io/projected/86a19396-fc1d-4247-bd55-b5e8f08d3623-kube-api-access-6cr8p\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:34.085371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.085347 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:34.085371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.085361 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a19396-fc1d-4247-bd55-b5e8f08d3623-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:34.322631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.322594 4959 generic.go:356] "Generic (PLEG): container finished" podID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerID="4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697" exitCode=0 Oct 30 20:15:34.322768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.322658 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" Oct 30 20:15:34.322768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.322756 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" event={"ID":"86a19396-fc1d-4247-bd55-b5e8f08d3623","Type":"ContainerDied","Data":"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697"} Oct 30 20:15:34.322878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.322795 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn" event={"ID":"86a19396-fc1d-4247-bd55-b5e8f08d3623","Type":"ContainerDied","Data":"2b36d97c87ec326810215561b26e5ae5bf41e05010feb95be62dff24bc6b423c"} Oct 30 20:15:34.322878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.322815 4959 scope.go:122] "RemoveContainer" containerID="4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697" Oct 30 20:15:34.324590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.324567 4959 generic.go:356] "Generic (PLEG): container finished" podID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerID="d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a" exitCode=0 Oct 30 20:15:34.324741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.324599 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" event={"ID":"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4","Type":"ContainerDied","Data":"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a"} Oct 30 20:15:34.324741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.324633 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" Oct 30 20:15:34.324741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.324678 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr" event={"ID":"bc6260a9-9d30-47fc-9348-5d2ca42b3bd4","Type":"ContainerDied","Data":"f10b64cfb9511e8bb6e23fd13e12ac95a0066e6500f6916877484ca0c83116da"} Oct 30 20:15:34.334718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.334700 4959 scope.go:122] "RemoveContainer" containerID="4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697" Oct 30 20:15:34.335011 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:34.334992 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697\": container with ID starting with 4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697 not found: ID does not exist" containerID="4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697" Oct 30 20:15:34.335086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.335027 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697"} err="failed to get container status \"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697\": rpc error: code = NotFound desc = could not find container \"4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697\": container with ID starting with 4a0e3b1502c0b76af5705110c4dd323add433982433f6d189e15e3b558325697 not found: ID does not exist" Oct 30 20:15:34.335086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.335044 4959 scope.go:122] "RemoveContainer" containerID="d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a" Oct 30 20:15:34.345701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.345676 4959 scope.go:122] "RemoveContainer" containerID="d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a" Oct 30 20:15:34.345961 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:34.345942 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a\": container with ID starting with d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a not found: ID does not exist" containerID="d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a" Oct 30 20:15:34.346030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.345968 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a"} err="failed to get container status \"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a\": rpc error: code = NotFound desc = could not find container \"d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a\": container with ID starting with d0eb2aaf2513265cc64b0e84f677b5e7ec1477ce69da19385efbd39e8b73930a not found: ID does not exist" Oct 30 20:15:34.375452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.375433 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:15:34.378844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.378823 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d72c-77ffd99dc9-sbzdr"] Oct 30 20:15:34.409034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.409016 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:15:34.415465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:34.415444 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d72c-8678cd5f84-w24rn"] Oct 30 20:15:35.371849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:35.371818 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:35.999234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:35.999193 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" path="/var/lib/kubelet/pods/86a19396-fc1d-4247-bd55-b5e8f08d3623/volumes" Oct 30 20:15:35.999697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:35.999669 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" path="/var/lib/kubelet/pods/bc6260a9-9d30-47fc-9348-5d2ca42b3bd4/volumes" Oct 30 20:15:37.106239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:37.106203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.232336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:38.232298 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:38.804050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:38.804007 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.104708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:40.104674 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:40.363781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:40.363696 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.122336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:41.122303 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.238032 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:41.237998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.345342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:41.345308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.781029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:41.780994 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.011347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:43.011305 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:44.711430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:44.711390 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.016845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.016818 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:15:45.123457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.123424 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\") pod \"70c04304-9a22-469b-8152-593ddecb4254\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " Oct 30 20:15:45.123660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.123601 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\" (UniqueName: \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\") pod \"70c04304-9a22-469b-8152-593ddecb4254\" (UID: \"70c04304-9a22-469b-8152-593ddecb4254\") " Oct 30 20:15:45.123908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.123887 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume" pod "70c04304-9a22-469b-8152-593ddecb4254" (UID: "70c04304-9a22-469b-8152-593ddecb4254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:45.124291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.124274 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c04304-9a22-469b-8152-593ddecb4254-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:45.125896 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.125875 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl" pod "70c04304-9a22-469b-8152-593ddecb4254" (UID: "70c04304-9a22-469b-8152-593ddecb4254"). InnerVolumeSpecName "kube-api-access-p8jwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:45.225486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.225452 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8jwl\" (UniqueName: \"kubernetes.io/projected/70c04304-9a22-469b-8152-593ddecb4254-kube-api-access-p8jwl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:45.414236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.414204 4959 generic.go:356] "Generic (PLEG): container finished" podID="70c04304-9a22-469b-8152-593ddecb4254" containerID="47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211" exitCode=0 Oct 30 20:15:45.414423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.414253 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" Oct 30 20:15:45.414423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.414286 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" event={"ID":"70c04304-9a22-469b-8152-593ddecb4254","Type":"ContainerDied","Data":"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211"} Oct 30 20:15:45.414423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.414347 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x" event={"ID":"70c04304-9a22-469b-8152-593ddecb4254","Type":"ContainerDied","Data":"2e6e79b9385b92d3f1ad470ccc52d26c2b6fdef1e4a2cc584c52b1ec21545f01"} Oct 30 20:15:45.414423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.414375 4959 scope.go:122] "RemoveContainer" containerID="47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211" Oct 30 20:15:45.426082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.426066 4959 scope.go:122] "RemoveContainer" containerID="47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211" Oct 30 20:15:45.426338 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:45.426320 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211\": container with ID starting with 47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211 not found: ID does not exist" containerID="47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211" Oct 30 20:15:45.426381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.426347 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211"} err="failed to get container status \"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211\": rpc error: code = NotFound desc = could not find container \"47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211\": container with ID starting with 47b0a7f9f2e4b81151420875c2d8b80f0a56178cd75fff426b81246f4f3db211 not found: ID does not exist" Oct 30 20:15:45.464172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.464136 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:15:45.474987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:45.474959 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6cd2ad-56c45f7fcf-w5m6x"] Oct 30 20:15:46.000719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:46.000686 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c04304-9a22-469b-8152-593ddecb4254" path="/var/lib/kubelet/pods/70c04304-9a22-469b-8152-593ddecb4254/volumes" Oct 30 20:15:46.067089 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:46.067053 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:15:46.415463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:46.415410 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.841269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:47.841232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.956167 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:47.956123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.188952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:48.188922 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.231456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:48.231423 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:48.296383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:48.296345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:49.698892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:49.698845 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:15:49.699447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:49.699091 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" podUID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" containerName="maas-api" containerID="cri-o://305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1" gracePeriod=30 Oct 30 20:15:49.774717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:49.774681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:49.908194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:49.908157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:49.988647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:49.988623 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:15:50.090509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.090468 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\") pod \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " Oct 30 20:15:50.090720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.090574 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\") pod \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " Oct 30 20:15:50.090720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.090693 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\") pod \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " Oct 30 20:15:50.090854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.090760 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\" (UniqueName: \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\") pod \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\" (UID: \"1b6deb66-12d8-46ad-b87c-4e24fba3a487\") " Oct 30 20:15:50.091107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.091076 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca" pod "1b6deb66-12d8-46ad-b87c-4e24fba3a487" (UID: "1b6deb66-12d8-46ad-b87c-4e24fba3a487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:50.091423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.091386 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6deb66-12d8-46ad-b87c-4e24fba3a487-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.092840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.092812 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls" pod "1b6deb66-12d8-46ad-b87c-4e24fba3a487" (UID: "1b6deb66-12d8-46ad-b87c-4e24fba3a487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:50.092965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.092897 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs" pod "1b6deb66-12d8-46ad-b87c-4e24fba3a487" (UID: "1b6deb66-12d8-46ad-b87c-4e24fba3a487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:50.093042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.093016 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2" pod "1b6deb66-12d8-46ad-b87c-4e24fba3a487" (UID: "1b6deb66-12d8-46ad-b87c-4e24fba3a487"). InnerVolumeSpecName "kube-api-access-6hmf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:50.193687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.193653 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmf2\" (UniqueName: \"kubernetes.io/projected/1b6deb66-12d8-46ad-b87c-4e24fba3a487-kube-api-access-6hmf2\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.193927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.193912 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.194073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.194060 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6deb66-12d8-46ad-b87c-4e24fba3a487-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.343898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.343767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.451670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.451639 4959 generic.go:356] "Generic (PLEG): container finished" podID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" containerID="305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1" exitCode=0 Oct 30 20:15:50.451875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.451707 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" Oct 30 20:15:50.451875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.451725 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" event={"ID":"1b6deb66-12d8-46ad-b87c-4e24fba3a487","Type":"ContainerDied","Data":"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1"} Oct 30 20:15:50.451875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.451781 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p" event={"ID":"1b6deb66-12d8-46ad-b87c-4e24fba3a487","Type":"ContainerDied","Data":"ad1ce0847b9ef08a746d44deb5a9a6c042a743fdc52623fbe1a01039837ee266"} Oct 30 20:15:50.451875 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.451808 4959 scope.go:122] "RemoveContainer" containerID="305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1" Oct 30 20:15:50.464123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.464104 4959 scope.go:122] "RemoveContainer" containerID="305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1" Oct 30 20:15:50.464424 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:50.464403 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1\": container with ID starting with 305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1 not found: ID does not exist" containerID="305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1" Oct 30 20:15:50.464477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.464434 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1"} err="failed to get container status \"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1\": rpc error: code = NotFound desc = could not find container \"305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1\": container with ID starting with 305dbce7971c29eeb3732e14bad1fd92f1064402da976ba7449d260151e0e4a1 not found: ID does not exist" Oct 30 20:15:50.502254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.502219 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:15:50.509818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.509790 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6cd2ad-5b58ffc7ff-9t47p"] Oct 30 20:15:50.611716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.611625 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.759701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.759669 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:15:50.805822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.805786 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\") pod \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " Oct 30 20:15:50.806065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.805975 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\" (UniqueName: \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\") pod \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\" (UID: \"36d7a7bc-f28b-47a0-a313-18cbe1d80262\") " Oct 30 20:15:50.806883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.806858 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume" pod "36d7a7bc-f28b-47a0-a313-18cbe1d80262" (UID: "36d7a7bc-f28b-47a0-a313-18cbe1d80262"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:50.825938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.810047 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td" pod "36d7a7bc-f28b-47a0-a313-18cbe1d80262" (UID: "36d7a7bc-f28b-47a0-a313-18cbe1d80262"). InnerVolumeSpecName "kube-api-access-tl6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:50.874288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.874189 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.908286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.908240 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6td\" (UniqueName: \"kubernetes.io/projected/36d7a7bc-f28b-47a0-a313-18cbe1d80262-kube-api-access-tl6td\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:50.908286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:50.908280 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d7a7bc-f28b-47a0-a313-18cbe1d80262-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:51.171872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.171792 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:51.485884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.485799 4959 generic.go:356] "Generic (PLEG): container finished" podID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerID="07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e" exitCode=0 Oct 30 20:15:51.485884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.485866 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" Oct 30 20:15:51.486074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.485888 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" event={"ID":"36d7a7bc-f28b-47a0-a313-18cbe1d80262","Type":"ContainerDied","Data":"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e"} Oct 30 20:15:51.486074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.485972 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh" event={"ID":"36d7a7bc-f28b-47a0-a313-18cbe1d80262","Type":"ContainerDied","Data":"b8ac53c30cfe34ebed0328a28c748e588b5e8c6fcbb9b3dc5006ac6a1b5423e5"} Oct 30 20:15:51.486074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.485993 4959 scope.go:122] "RemoveContainer" containerID="07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e" Oct 30 20:15:51.498056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.498038 4959 scope.go:122] "RemoveContainer" containerID="07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e" Oct 30 20:15:51.498403 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:51.498383 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e\": container with ID starting with 07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e not found: ID does not exist" containerID="07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e" Oct 30 20:15:51.498443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.498416 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e"} err="failed to get container status \"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e\": rpc error: code = NotFound desc = could not find container \"07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e\": container with ID starting with 07b3dad5a704ed6a7cbe726a093cb9670a2ad65f1adfa598275036ca5d2c087e not found: ID does not exist" Oct 30 20:15:51.539562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.539526 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:15:51.550728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.550700 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4a31d3-ff77d8dc-ccnfh"] Oct 30 20:15:51.999753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:51.999720 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" path="/var/lib/kubelet/pods/1b6deb66-12d8-46ad-b87c-4e24fba3a487/volumes" Oct 30 20:15:52.000218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:52.000201 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" path="/var/lib/kubelet/pods/36d7a7bc-f28b-47a0-a313-18cbe1d80262/volumes" Oct 30 20:15:52.797519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:52.797059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.400150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:54.400112 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:56.020254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:56.020220 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.447370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.447335 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:15:57.447906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.447649 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" podUID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" containerName="maas-api" containerID="cri-o://c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560" gracePeriod=30 Oct 30 20:15:57.516204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.516165 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.822537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.821305 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:15:57.907217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.905771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.911269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.911241 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\") pod \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " Oct 30 20:15:57.911448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.911282 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\") pod \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " Oct 30 20:15:57.911448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.911318 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\") pod \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " Oct 30 20:15:57.911448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.911434 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\" (UniqueName: \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\") pod \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\" (UID: \"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0\") " Oct 30 20:15:57.911742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.911719 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca" pod "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" (UID: "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:57.912136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.912111 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:57.914778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.914748 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls" pod "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" (UID: "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:57.914999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.914964 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv" pod "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" (UID: "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0"). InnerVolumeSpecName "kube-api-access-z92jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:57.915376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:57.915322 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs" pod "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" (UID: "c0cb5480-96e2-4a7e-8c61-78fb5e3431c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:58.018574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.018479 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:58.018574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.018531 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:58.018574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.018546 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z92jv\" (UniqueName: \"kubernetes.io/projected/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0-kube-api-access-z92jv\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:15:58.024620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.024041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.228057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.228022 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.340527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.339964 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.457658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.457629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.538786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.538749 4959 generic.go:356] "Generic (PLEG): container finished" podID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" containerID="c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560" exitCode=0 Oct 30 20:15:58.538937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.538788 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" event={"ID":"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0","Type":"ContainerDied","Data":"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560"} Oct 30 20:15:58.538937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.538831 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" Oct 30 20:15:58.538937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.538857 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn" event={"ID":"c0cb5480-96e2-4a7e-8c61-78fb5e3431c0","Type":"ContainerDied","Data":"449cb98baf7d3cc95d7474c2142bb9c198144980129c25bf376f9d9b7fe9f627"} Oct 30 20:15:58.538937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.538886 4959 scope.go:122] "RemoveContainer" containerID="c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560" Oct 30 20:15:58.551206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.550556 4959 scope.go:122] "RemoveContainer" containerID="c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560" Oct 30 20:15:58.551206 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:15:58.551140 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560\": container with ID starting with c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560 not found: ID does not exist" containerID="c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560" Oct 30 20:15:58.551206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.551183 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560"} err="failed to get container status \"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560\": rpc error: code = NotFound desc = could not find container \"c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560\": container with ID starting with c1d098af65bd48ac4cc948e09ce13e512c1ed1fc554c8ba1f43480391b513560 not found: ID does not exist" Oct 30 20:15:58.587544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.587511 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:15:58.599030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.598947 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4a31d3-56946d7c69-qmbxn"] Oct 30 20:15:58.836380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:58.836345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.080128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.080095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.188572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.188541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.472702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.472663 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:15:59.473840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473814 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.473840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473840 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473869 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2ac7786e-fbd5-4d81-af38-1b336196bd05" containerName="payload-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473877 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac7786e-fbd5-4d81-af38-1b336196bd05" containerName="payload-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473896 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473902 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473923 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473930 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473947 4959 cpu_manager.go:414] "containerMap: removing container" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473953 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473966 4959 cpu_manager.go:414] "containerMap: removing container" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473973 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473989 4959 cpu_manager.go:414] "containerMap: removing container" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.473996 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474009 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" containerName="maas-api" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474016 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" containerName="maas-api" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474032 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474039 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" Oct 30 20:15:59.474046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474053 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474060 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474074 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474081 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474095 4959 cpu_manager.go:414] "containerMap: removing container" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474102 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474112 4959 cpu_manager.go:414] "containerMap: removing container" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474119 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474130 4959 cpu_manager.go:414] "containerMap: removing container" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474137 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474149 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474155 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474167 4959 cpu_manager.go:414] "containerMap: removing container" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474176 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474186 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474193 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474202 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474208 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474218 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474224 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474238 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" containerName="maas-api" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474245 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" containerName="maas-api" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474261 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474267 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474280 4959 cpu_manager.go:414] "containerMap: removing container" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474287 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474300 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474306 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474317 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474323 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474475 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474488 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474520 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474532 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d7a7bc-f28b-47a0-a313-18cbe1d80262" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474544 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" containerName="maas-api" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474553 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c3d7f6-14af-433c-9db2-553060cfa7cc" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474563 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474571 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474581 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="740a4953-6deb-48e4-90cd-b5d09c1b9706" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474589 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c04304-9a22-469b-8152-593ddecb4254" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474599 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474606 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474617 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474626 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474633 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474643 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474650 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6deb66-12d8-46ad-b87c-4e24fba3a487" containerName="maas-api" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474659 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a19396-fc1d-4247-bd55-b5e8f08d3623" containerName="payload-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474669 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6260a9-9d30-47fc-9348-5d2ca42b3bd4" containerName="payload-pre-processing" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474678 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474688 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce4252b-e340-49c1-953a-33e6abb94ac2" containerName="cleanup" Oct 30 20:15:59.474907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.474699 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac7786e-fbd5-4d81-af38-1b336196bd05" containerName="payload-processing" Oct 30 20:15:59.479638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.479616 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.502755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.502654 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:15:59.508295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.508261 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-77ac5a-openshift-default-dockercfg-497sf\"" Oct 30 20:15:59.529979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.529950 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.643745 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.643810 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.643855 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.643905 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.643965 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.644082 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.644107 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.644182 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.648879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.644286 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk27l\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.690277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.690245 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.746206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746113 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746152 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746205 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746287 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk27l\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746331 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746406 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746448 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746546 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746595 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.746910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.746876 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.747079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.747059 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.747814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.747334 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.747814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.747426 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.750531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.749534 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.750756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.750732 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.764907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.764885 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk27l\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.765013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.764885 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\") pod \"e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:15:59.798938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:15:59.798905 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:16:00.000453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.000372 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0cb5480-96e2-4a7e-8c61-78fb5e3431c0" path="/var/lib/kubelet/pods/c0cb5480-96e2-4a7e-8c61-78fb5e3431c0/volumes" Oct 30 20:16:00.139260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.139228 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:16:00.143165 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:16:00.143133 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bfdb23_7483_4676_ac5b_36a7cde42d55.slice/crio-8c4fd918bd658dcf95c635ddf1a500153950f8253381bb5b3838439dcb1c9f3f WatchSource:0}: Error finding container 8c4fd918bd658dcf95c635ddf1a500153950f8253381bb5b3838439dcb1c9f3f: Status 404 returned error can't find the container with id 8c4fd918bd658dcf95c635ddf1a500153950f8253381bb5b3838439dcb1c9f3f Oct 30 20:16:00.145994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.145957 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:16:00.146086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.146030 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:16:00.146086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.146061 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:16:00.558580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.558542 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerStarted","Data":"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68"} Oct 30 20:16:00.559004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.558610 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerStarted","Data":"8c4fd918bd658dcf95c635ddf1a500153950f8253381bb5b3838439dcb1c9f3f"} Oct 30 20:16:00.623090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.623034 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podStartSLOduration=1.6230186450000001 podStartE2EDuration="1.623018645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:00.609108672 +0000 UTC m=+2484.738342593" watchObservedRunningTime="2026-10-30 20:16:00.623018645 +0000 UTC m=+2484.752252624" Oct 30 20:16:00.799404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:00.799367 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:16:01.092739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:01.092709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:01.805326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:01.805299 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:16:02.574675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:02.574645 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:16:02.575585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:02.575567 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:16:03.079664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:03.079630 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:03.187141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:03.187099 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:04.790137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:04.790097 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:04.929020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:04.928992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:05.177878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:05.177840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.329388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:06.329347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.433365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:06.433324 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.655924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:06.655891 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:16:06.656196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:06.656163 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" containerID="cri-o://1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d" gracePeriod=30 Oct 30 20:16:06.679626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:06.679598 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:07.030465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:07.030384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:07.483795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:07.483761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:07.987360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:07.987317 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:08.132541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:08.132453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:08.849346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:08.849305 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:10.604156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:10.604116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:11.934613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.934550 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:16:11.991041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991014 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991100 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991254 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991302 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991355 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991397 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991450 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991483 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991535 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\") pod \"e430d411-2526-4b02-8452-d9eb1fbe99fb\" (UID: \"e430d411-2526-4b02-8452-d9eb1fbe99fb\") " Oct 30 20:16:11.991913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.991808 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:11.992540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.992484 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:11.992844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.992820 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:11.992935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.992844 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:11.993345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.993320 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:11.993539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.993351 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:11.993539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.993364 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:11.993539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.993376 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:11.994940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.994021 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:11.995616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.995579 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:11.996282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.996253 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "kube-api-access-8jskp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:11.996536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.996510 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:11.998881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:11.998857 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token" pod "e430d411-2526-4b02-8452-d9eb1fbe99fb" (UID: "e430d411-2526-4b02-8452-d9eb1fbe99fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:12.095328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.095303 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jskp\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-kube-api-access-8jskp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:12.095328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.095328 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:12.095542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.095339 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e430d411-2526-4b02-8452-d9eb1fbe99fb-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:12.095542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.095348 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:12.095542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.095355 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e430d411-2526-4b02-8452-d9eb1fbe99fb-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:12.125017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.124989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.252698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.252612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.364481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.364439 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.480129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.480085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.650859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.647751 4959 generic.go:356] "Generic (PLEG): container finished" podID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerID="1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d" exitCode=0 Oct 30 20:16:12.650859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.647831 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerDied","Data":"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d"} Oct 30 20:16:12.650859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.647845 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" Oct 30 20:16:12.650859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.647920 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92" event={"ID":"e430d411-2526-4b02-8452-d9eb1fbe99fb","Type":"ContainerDied","Data":"a007d54a7a12a92c4077dbf829d1e92bf7e678398f29f797602c2af9c12144b1"} Oct 30 20:16:12.650859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.647945 4959 scope.go:122] "RemoveContainer" containerID="1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d" Oct 30 20:16:12.659746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.659729 4959 scope.go:122] "RemoveContainer" containerID="dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780" Oct 30 20:16:12.672585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.672563 4959 scope.go:122] "RemoveContainer" containerID="1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d" Oct 30 20:16:12.672977 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:12.672951 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d\": container with ID starting with 1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d not found: ID does not exist" containerID="1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d" Oct 30 20:16:12.673059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.672994 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d"} err="failed to get container status \"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d\": rpc error: code = NotFound desc = could not find container \"1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d\": container with ID starting with 1f3f01d3a8c534fdcf607f5d727a00fd6f0cbaacafa8c9cefe28bc936b3e9a4d not found: ID does not exist" Oct 30 20:16:12.673059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.673016 4959 scope.go:122] "RemoveContainer" containerID="dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780" Oct 30 20:16:12.673296 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:12.673267 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780\": container with ID starting with dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780 not found: ID does not exist" containerID="dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780" Oct 30 20:16:12.673353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.673306 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780"} err="failed to get container status \"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780\": rpc error: code = NotFound desc = could not find container \"dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780\": container with ID starting with dc418308b18b00b118c08aa7f123266c35559e7e710590c1a200e6ac493ba780 not found: ID does not exist" Oct 30 20:16:12.743226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.743194 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:16:12.757278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:12.757250 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4a31d3-openshift-default-5c6ff697b5-qhd92"] Oct 30 20:16:13.890567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:13.890530 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.000394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.000364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.000610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.000521 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" path="/var/lib/kubelet/pods/e430d411-2526-4b02-8452-d9eb1fbe99fb/volumes" Oct 30 20:16:14.000967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.000951 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:16:14.001870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001855 4959 cpu_manager.go:414] "containerMap: removing container" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:16:14.001922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001874 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:16:14.001922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001911 4959 cpu_manager.go:414] "containerMap: removing container" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:16:14.001922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001918 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:16:14.002022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001934 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:16:14.002022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001942 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:16:14.002022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001958 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:14.002022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.001965 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:14.002146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002116 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb83428-6848-477e-a30f-e002ed82a582" containerName="cleanup" Oct 30 20:16:14.002146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002127 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ced1f3-94be-4cf7-ad4d-0014ce2cfde3" containerName="cleanup" Oct 30 20:16:14.002146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002141 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fe9a83-17bb-4325-a1ed-b2c3537e04af" containerName="cleanup" Oct 30 20:16:14.002241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002149 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1d3ed7-96ce-4dc3-b8e9-03c0932c2054" containerName="cleanup" Oct 30 20:16:14.002241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002161 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a400f76-2181-4479-ac00-a2da0716333d" containerName="cleanup" Oct 30 20:16:14.002241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002171 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:14.002241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.002181 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:14.007752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.007725 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.012014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.011896 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:16:14.033118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.033090 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5314c2\"" Oct 30 20:16:14.033118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.033102 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5314c2-dockercfg-zh2qq\"" Oct 30 20:16:14.137519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.134427 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwll\" (UniqueName: \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.137519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.134726 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.235874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.235791 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwll\" (UniqueName: \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.236025 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.235992 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.236769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.236741 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.237300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.237272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.254600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.254563 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwll\" (UniqueName: \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\") pod \"payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.325330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.325243 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:14.638016 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.637982 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:16:14.641450 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:16:14.641420 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bea1573_2af5_4655_8184_c008a3f4659b.slice/crio-f888c3eebeccdbfaaad969fbfebd02df73cfe1ed01c0ae9136f46272a81c70c9 WatchSource:0}: Error finding container f888c3eebeccdbfaaad969fbfebd02df73cfe1ed01c0ae9136f46272a81c70c9: Status 404 returned error can't find the container with id f888c3eebeccdbfaaad969fbfebd02df73cfe1ed01c0ae9136f46272a81c70c9 Oct 30 20:16:14.665420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:14.665393 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" event={"ID":"0bea1573-2af5-4655-8184-c008a3f4659b","Type":"ContainerStarted","Data":"f888c3eebeccdbfaaad969fbfebd02df73cfe1ed01c0ae9136f46272a81c70c9"} Oct 30 20:16:15.673309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:15.673275 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" event={"ID":"0bea1573-2af5-4655-8184-c008a3f4659b","Type":"ContainerStarted","Data":"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7"} Oct 30 20:16:15.673850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:15.673337 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:15.712724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:15.712676 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" podStartSLOduration=2.712663954 podStartE2EDuration="2.712663954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:15.712229879 +0000 UTC m=+2499.841463799" watchObservedRunningTime="2026-10-30 20:16:15.712663954 +0000 UTC m=+2499.841897866" Oct 30 20:16:16.024274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:16.024190 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:16.087445 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:16.087403 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:16:17.690824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:17.690788 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:19.423526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:19.423480 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:21.128705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:21.128674 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:22.801001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:22.800955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:24.540777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:24.540745 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.038587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:26.037963 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:16:26.067598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:26.067540 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.684074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:26.684044 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:16:26.739644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:26.739042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:28.511008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:28.510971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.324189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:30.324153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.082272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:32.082233 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.188394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:32.188340 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.476205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:32.476153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:33.842175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:33.842138 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.624330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:35.624295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:37.388851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:37.388814 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.246298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:39.246253 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.353086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:39.353052 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:41.007488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.007453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:41.077520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.075894 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" podUID="7dc49ab8-dc5a-45ec-876e-461391f06947" containerName="main" containerID="cri-o://1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545" gracePeriod=45 Oct 30 20:16:41.365824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.365797 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:16:41.425013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.424978 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425055 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425144 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\" (UniqueName: \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425188 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425215 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425292 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\") pod \"7dc49ab8-dc5a-45ec-876e-461391f06947\" (UID: \"7dc49ab8-dc5a-45ec-876e-461391f06947\") " Oct 30 20:16:41.425447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425360 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:41.425680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425619 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:41.425739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.425709 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:41.426137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.426117 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.426181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.426145 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.426181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.426158 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.428255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.428228 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:41.428372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.428279 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "kube-api-access-82xvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:41.428437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.428378 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs" pod "7dc49ab8-dc5a-45ec-876e-461391f06947" (UID: "7dc49ab8-dc5a-45ec-876e-461391f06947"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:41.527706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.527667 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82xvm\" (UniqueName: \"kubernetes.io/projected/7dc49ab8-dc5a-45ec-876e-461391f06947-kube-api-access-82xvm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.527706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.527703 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc49ab8-dc5a-45ec-876e-461391f06947-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.527706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.527713 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc49ab8-dc5a-45ec-876e-461391f06947-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:16:41.870040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.870009 4959 generic.go:356] "Generic (PLEG): container finished" podID="7dc49ab8-dc5a-45ec-876e-461391f06947" containerID="1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545" exitCode=0 Oct 30 20:16:41.870271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.870061 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" Oct 30 20:16:41.870271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.870095 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" event={"ID":"7dc49ab8-dc5a-45ec-876e-461391f06947","Type":"ContainerDied","Data":"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545"} Oct 30 20:16:41.870271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.870146 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4" event={"ID":"7dc49ab8-dc5a-45ec-876e-461391f06947","Type":"ContainerDied","Data":"7f9e5839590f685b3c14c1f6bd0af7880c594b44b55e4ae113f694b75ea94ee8"} Oct 30 20:16:41.870271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.870168 4959 scope.go:122] "RemoveContainer" containerID="1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545" Oct 30 20:16:41.881637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.881618 4959 scope.go:122] "RemoveContainer" containerID="1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545" Oct 30 20:16:41.881950 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:41.881903 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545\": container with ID starting with 1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545 not found: ID does not exist" containerID="1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545" Oct 30 20:16:41.881950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.881933 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545"} err="failed to get container status \"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545\": rpc error: code = NotFound desc = could not find container \"1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545\": container with ID starting with 1ed503f963decd93273f51e936122efea42f7984b63c17bd07b1a6917bf78545 not found: ID does not exist" Oct 30 20:16:41.918947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.918922 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:16:41.929865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:41.929834 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c8fc40/test-model-c8fc40-kserve-54fb88954c-726w4"] Oct 30 20:16:42.000084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:42.000054 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc49ab8-dc5a-45ec-876e-461391f06947" path="/var/lib/kubelet/pods/7dc49ab8-dc5a-45ec-876e-461391f06947/volumes" Oct 30 20:16:42.675152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:42.675119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.560796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:44.560759 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:46.067554 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:46.067476 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:16:46.180550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:46.180453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.890783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:47.890749 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:49.478737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.478703 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:16:49.479440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.479423 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7dc49ab8-dc5a-45ec-876e-461391f06947" containerName="main" Oct 30 20:16:49.479487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.479441 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc49ab8-dc5a-45ec-876e-461391f06947" containerName="main" Oct 30 20:16:49.479487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.479472 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:49.479487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.479477 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e430d411-2526-4b02-8452-d9eb1fbe99fb" containerName="istio-proxy" Oct 30 20:16:49.479651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.479583 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc49ab8-dc5a-45ec-876e-461391f06947" containerName="main" Oct 30 20:16:49.484813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.484796 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.503330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.503294 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:16:49.509853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.509823 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5314c2-serving-cert\"" Oct 30 20:16:49.536118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.536086 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.536303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.536284 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.536348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.536332 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.536472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.536448 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmzr\" (UniqueName: \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.570047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.570014 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:49.637297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.637262 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.637512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.637377 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.637512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.637411 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.637512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.637439 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmzr\" (UniqueName: \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.637716 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:49.637563 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-serving-cert: secret "maas-api-e2e-shared-a-w0-5314c2-serving-cert" not found Oct 30 20:16:49.637716 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:16:49.637640 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls podName:a8df0448-d3cb-4d7c-8315-bbfeb8206530 nodeName:}" failed. No retries permitted until 2026-10-30 20:16:50.137620884 +0000 UTC m=+2534.266854797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls") pod "maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" (UID: "a8df0448-d3cb-4d7c-8315-bbfeb8206530") : secret "maas-api-e2e-shared-a-w0-5314c2-serving-cert" not found Oct 30 20:16:49.638070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.638047 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.640053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.640033 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.656126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.656100 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmzr\" (UniqueName: \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:49.678267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.678239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:49.787998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:49.787899 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:50.145624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.145597 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:50.148287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.148255 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:50.399026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.398946 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:50.740685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.740601 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:16:50.744678 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:16:50.744646 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8df0448_d3cb_4d7c_8315_bbfeb8206530.slice/crio-1bf6b266893de4e15216655a455386cd644eb532548d4375c1de7ff822c1b45d WatchSource:0}: Error finding container 1bf6b266893de4e15216655a455386cd644eb532548d4375c1de7ff822c1b45d: Status 404 returned error can't find the container with id 1bf6b266893de4e15216655a455386cd644eb532548d4375c1de7ff822c1b45d Oct 30 20:16:50.746082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.746060 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:16:50.943213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:50.943180 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" event={"ID":"a8df0448-d3cb-4d7c-8315-bbfeb8206530","Type":"ContainerStarted","Data":"1bf6b266893de4e15216655a455386cd644eb532548d4375c1de7ff822c1b45d"} Oct 30 20:16:51.497114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:51.497084 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.951717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:51.951682 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" event={"ID":"a8df0448-d3cb-4d7c-8315-bbfeb8206530","Type":"ContainerStarted","Data":"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158"} Oct 30 20:16:51.952155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:51.951749 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:52.002998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:52.002949 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" podStartSLOduration=2.705188193 podStartE2EDuration="3.002933658s" totalImagesPullingTime="297.745465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:52.001965397 +0000 UTC m=+2536.131199319" watchObservedRunningTime="2026-10-30 20:16:52.002933658 +0000 UTC m=+2536.132167586" Oct 30 20:16:53.239995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:53.239961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.892230 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:54.892182 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:56.636101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:56.636069 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:57.964309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:57.964268 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:16:58.408648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:58.408572 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:58.518716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:58.518685 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:59.750169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:16:59.750131 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:00.180647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:00.180613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:00.673209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:00.673174 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:17:00.709398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:00.709366 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.212846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:02.212811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:03.641764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:03.641722 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:04.006288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:04.006210 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:05.739594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:05.739560 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:07.498072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:07.498040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.182670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:09.182637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.014859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:11.014828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.317447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:11.317352 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:17:11.419372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:11.419332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.534385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:11.534348 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:12.791022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:12.790977 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.407700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:14.407649 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:15.695678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:15.695633 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" podUID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" containerName="main" containerID="cri-o://0617611d85df381290f39456b407683d0662d42d4d66a0d4abc9170461312cc2" gracePeriod=45 Oct 30 20:17:16.099619 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:17:16.099560 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:17:16.150848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.150814 4959 generic.go:356] "Generic (PLEG): container finished" podID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" containerID="0617611d85df381290f39456b407683d0662d42d4d66a0d4abc9170461312cc2" exitCode=0 Oct 30 20:17:16.151021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.150953 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" event={"ID":"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6","Type":"ContainerDied","Data":"0617611d85df381290f39456b407683d0662d42d4d66a0d4abc9170461312cc2"} Oct 30 20:17:16.151021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.150992 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" event={"ID":"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6","Type":"ContainerDied","Data":"de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518"} Oct 30 20:17:16.151021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.151008 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0208eff87f7685eafc833bc1842271acfeb1fc4c596aab4f2253ba65ae5518" Oct 30 20:17:16.166858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.166834 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:17:16.300646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300515 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\" (UniqueName: \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.300646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300575 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.300646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300599 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.300943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300798 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.300943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300828 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.300943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300883 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\") pod \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\" (UID: \"1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6\") " Oct 30 20:17:16.301097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.300942 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:16.301187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.301158 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:16.301270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.301244 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:16.301449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.301430 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.301558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.301455 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.301558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.301469 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.303187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.303160 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "kube-api-access-bl2vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:16.303486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.303466 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:16.303601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.303554 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs" pod "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" (UID: "1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:16.322453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.320196 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" containerID="cri-o://9630128a3abba7ed84237e04425804d51c511706f4863ea82f2f1fbb8902f7f3" gracePeriod=25 Oct 30 20:17:16.402590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.402550 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2vt\" (UniqueName: \"kubernetes.io/projected/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-kube-api-access-bl2vt\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.402590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.402579 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.402590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.402588 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:16.573160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:16.573071 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:17.190808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:17.190766 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m" Oct 30 20:17:17.273059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:17.273018 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:17:17.295768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:17.287749 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d3c508/test-model-d3c508-kserve-58f9c95bc7-58t9m"] Oct 30 20:17:18.001277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:18.001237 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" path="/var/lib/kubelet/pods/1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6/volumes" Oct 30 20:17:18.614407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:18.614363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:20.716730 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:20.716651 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:21.103468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:21.103341 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:21.773562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:21.773467 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:22.281799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:22.280124 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:22.499943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:22.499902 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:22.698087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:22.698053 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:23.014560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:23.014526 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.856696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:24.856661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.968651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:24.968618 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.355368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:25.355333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:26.042206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.042175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:26.154048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.154018 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:26.320160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.320077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:26.435112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.435075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:26.464952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.464921 4959 generic.go:356] "Generic (PLEG): container finished" podID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerID="01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68" exitCode=0 Oct 30 20:17:26.465105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.464980 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerDied","Data":"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68"} Oct 30 20:17:26.465457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.465441 4959 scope.go:122] "RemoveContainer" containerID="01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68" Oct 30 20:17:26.467262 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.467235 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:17:26.467326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.467305 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:17:26.467364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.467338 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:17:26.572780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:26.571932 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:27.473421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.473387 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerStarted","Data":"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2"} Oct 30 20:17:27.556777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.556729 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:27.804435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.804344 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:17:27.804683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.804607 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:27.804764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.804683 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:27.985910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:27.985875 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:28.085425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.085351 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:17:28.086109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.086093 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" containerName="main" Oct 30 20:17:28.086160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.086112 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" containerName="main" Oct 30 20:17:28.086224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.086215 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5d34fc-9370-46ea-98ec-d3ee86d8cbf6" containerName="main" Oct 30 20:17:28.089397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.089380 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.091727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.091697 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:28.104001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.103977 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-188d45\"" Oct 30 20:17:28.104127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.104099 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-188d45-dockercfg-vbdl2\"" Oct 30 20:17:28.107922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.107896 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:17:28.177522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.147786 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:17:28.182684 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.181776 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvg7z\" (UniqueName: \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.182684 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.181856 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.184055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.184029 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:17:28.185571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.185332 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.293192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.293062 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tj4\" (UniqueName: \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.293466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.293276 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.293466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.293408 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvg7z\" (UniqueName: \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.293466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.293441 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.294543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.294471 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.340653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.338246 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvg7z\" (UniqueName: \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\") pod \"payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.395202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.395164 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tj4\" (UniqueName: \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.395393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.395221 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.396201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.396177 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.403163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.403138 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:28.422981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.422737 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tj4\" (UniqueName: \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\") pod \"payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.527715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.527681 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:28.777519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.776431 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:17:28.782990 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:17:28.782962 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a974811_ce19_4073_8599_89d75aa07947.slice/crio-aff1f2582bd2e69443ea519ee54bbf759866e3bea463e1491afa7d4eed0c6694 WatchSource:0}: Error finding container aff1f2582bd2e69443ea519ee54bbf759866e3bea463e1491afa7d4eed0c6694: Status 404 returned error can't find the container with id aff1f2582bd2e69443ea519ee54bbf759866e3bea463e1491afa7d4eed0c6694 Oct 30 20:17:28.799661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.799617 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:28.799816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.799693 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:28.889151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:28.889055 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:17:28.893024 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:17:28.892989 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7b2a8c_79dd_4659_a2a6_faf932938ae0.slice/crio-a3ee7e2a6a166d07000a2b877f55332fc133aa4da1b88cc5dcd1e2c0ad483f5d WatchSource:0}: Error finding container a3ee7e2a6a166d07000a2b877f55332fc133aa4da1b88cc5dcd1e2c0ad483f5d: Status 404 returned error can't find the container with id a3ee7e2a6a166d07000a2b877f55332fc133aa4da1b88cc5dcd1e2c0ad483f5d Oct 30 20:17:29.494311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.494273 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" event={"ID":"3a974811-ce19-4073-8599-89d75aa07947","Type":"ContainerStarted","Data":"f7c2ab2f38db94b97e1e91904f79048f11c90856b86c4aae190b6185af5ced5f"} Oct 30 20:17:29.494535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.494371 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" event={"ID":"3a974811-ce19-4073-8599-89d75aa07947","Type":"ContainerStarted","Data":"aff1f2582bd2e69443ea519ee54bbf759866e3bea463e1491afa7d4eed0c6694"} Oct 30 20:17:29.494535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.494519 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:29.495856 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.495830 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" event={"ID":"5d7b2a8c-79dd-4659-a2a6-faf932938ae0","Type":"ContainerStarted","Data":"76649f05ced98ab06b2134dd7c685539cfc0ecde0a51851d90fdf2cb775afc1f"} Oct 30 20:17:29.495974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.495876 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" event={"ID":"5d7b2a8c-79dd-4659-a2a6-faf932938ae0","Type":"ContainerStarted","Data":"a3ee7e2a6a166d07000a2b877f55332fc133aa4da1b88cc5dcd1e2c0ad483f5d"} Oct 30 20:17:29.495974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.495895 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:29.531463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.531411 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" podStartSLOduration=1.531398598 podStartE2EDuration="1.531398598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:29.531315374 +0000 UTC m=+2573.660549295" watchObservedRunningTime="2026-10-30 20:17:29.531398598 +0000 UTC m=+2573.660632516" Oct 30 20:17:29.573252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.573186 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" podStartSLOduration=1.573167897 podStartE2EDuration="1.573167897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:29.573076469 +0000 UTC m=+2573.702310392" watchObservedRunningTime="2026-10-30 20:17:29.573167897 +0000 UTC m=+2573.702401819" Oct 30 20:17:29.799561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.799457 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:29.799729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.799552 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:29.918057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:29.918023 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:30.177078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:30.177017 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:30.288143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:30.288108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:30.799607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:30.799570 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:30.800039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:30.799646 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:31.800068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:31.800033 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:31.800596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:31.800097 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:31.899767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:31.899738 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:32.010643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.010604 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:32.078425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.078351 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:17:32.082906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.082888 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.100171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.100140 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:17:32.106645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.106622 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9d2720\"" Oct 30 20:17:32.106813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.106709 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9d2720-dockercfg-l8b47\"" Oct 30 20:17:32.216520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.204032 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.216520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.204122 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxfp\" (UniqueName: \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.246524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.242935 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:17:32.253826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.253795 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.311963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.311929 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:32.314430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.314338 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxfp\" (UniqueName: \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.315642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.315616 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.316043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.316021 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.317533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.317508 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzzw\" (UniqueName: \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.317785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.317428 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:17:32.321432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.316938 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.341547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.341458 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxfp\" (UniqueName: \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\") pod \"payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.399024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.398995 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:32.419163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.419129 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzzw\" (UniqueName: \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.419312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.419275 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.420060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.420038 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.439620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.439594 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzzw\" (UniqueName: \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\") pod \"payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.577530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.575738 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:17:32.634918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.634889 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:32.698219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.698183 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:32.734807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.734780 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:17:32.737306 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:17:32.737230 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9202623e_d88f_4f90_b72d_92cd8716eddf.slice/crio-e79ff40361985e2fa920d3b88e6cd6d16177e17b6d24dbfc5584fb4b290d3f49 WatchSource:0}: Error finding container e79ff40361985e2fa920d3b88e6cd6d16177e17b6d24dbfc5584fb4b290d3f49: Status 404 returned error can't find the container with id e79ff40361985e2fa920d3b88e6cd6d16177e17b6d24dbfc5584fb4b290d3f49 Oct 30 20:17:32.802564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.799445 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 20:17:32.802564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.799507 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 20:17:32.981274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:32.981244 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:17:32.982315 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:17:32.982283 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd16a23_dd26_4487_805c_598bd2cee8ff.slice/crio-1f58bb011885d1af5172ed7946a06446f5dad2cedcbb30ed929a60c5f3975a34 WatchSource:0}: Error finding container 1f58bb011885d1af5172ed7946a06446f5dad2cedcbb30ed929a60c5f3975a34: Status 404 returned error can't find the container with id 1f58bb011885d1af5172ed7946a06446f5dad2cedcbb30ed929a60c5f3975a34 Oct 30 20:17:33.559010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.558969 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" event={"ID":"0dd16a23-dd26-4487-805c-598bd2cee8ff","Type":"ContainerStarted","Data":"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb"} Oct 30 20:17:33.559349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.559326 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" event={"ID":"0dd16a23-dd26-4487-805c-598bd2cee8ff","Type":"ContainerStarted","Data":"1f58bb011885d1af5172ed7946a06446f5dad2cedcbb30ed929a60c5f3975a34"} Oct 30 20:17:33.560695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.560671 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:33.591516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.582773 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" event={"ID":"9202623e-d88f-4f90-b72d-92cd8716eddf","Type":"ContainerStarted","Data":"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db"} Oct 30 20:17:33.591516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.582847 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" event={"ID":"9202623e-d88f-4f90-b72d-92cd8716eddf","Type":"ContainerStarted","Data":"e79ff40361985e2fa920d3b88e6cd6d16177e17b6d24dbfc5584fb4b290d3f49"} Oct 30 20:17:33.591516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.583845 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:33.601047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.600988 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" podStartSLOduration=1.600970825 podStartE2EDuration="1.600970825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:33.598691598 +0000 UTC m=+2577.727925519" watchObservedRunningTime="2026-10-30 20:17:33.600970825 +0000 UTC m=+2577.730204746" Oct 30 20:17:33.640205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.640155 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" podStartSLOduration=1.640140536 podStartE2EDuration="1.640140536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:33.636264981 +0000 UTC m=+2577.765498901" watchObservedRunningTime="2026-10-30 20:17:33.640140536 +0000 UTC m=+2577.769374456" Oct 30 20:17:33.804091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.804059 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:17:33.805319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:33.805299 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:17:34.005962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:34.005929 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.004247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.004219 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.606133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.606101 4959 generic.go:356] "Generic (PLEG): container finished" podID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerID="9630128a3abba7ed84237e04425804d51c511706f4863ea82f2f1fbb8902f7f3" exitCode=0 Oct 30 20:17:36.606301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.606176 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" event={"ID":"b925d0cb-a3cd-452e-9742-07d79a2928d0","Type":"ContainerDied","Data":"9630128a3abba7ed84237e04425804d51c511706f4863ea82f2f1fbb8902f7f3"} Oct 30 20:17:36.606301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.606227 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" event={"ID":"b925d0cb-a3cd-452e-9742-07d79a2928d0","Type":"ContainerDied","Data":"1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc"} Oct 30 20:17:36.606301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.606247 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1acdd58fba2f554d7d7b7a90b67866328026d59c5665f09092d497c71419addc" Oct 30 20:17:36.612584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.612561 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:17:36.682647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.682614 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\") pod \"b925d0cb-a3cd-452e-9742-07d79a2928d0\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " Oct 30 20:17:36.682852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.682742 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\" (UniqueName: \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\") pod \"b925d0cb-a3cd-452e-9742-07d79a2928d0\" (UID: \"b925d0cb-a3cd-452e-9742-07d79a2928d0\") " Oct 30 20:17:36.683089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.683063 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume" pod "b925d0cb-a3cd-452e-9742-07d79a2928d0" (UID: "b925d0cb-a3cd-452e-9742-07d79a2928d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:36.683373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.683353 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b925d0cb-a3cd-452e-9742-07d79a2928d0-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:36.685328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.685303 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7" pod "b925d0cb-a3cd-452e-9742-07d79a2928d0" (UID: "b925d0cb-a3cd-452e-9742-07d79a2928d0"). InnerVolumeSpecName "kube-api-access-w8zj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:36.784671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:36.784639 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zj7\" (UniqueName: \"kubernetes.io/projected/b925d0cb-a3cd-452e-9742-07d79a2928d0-kube-api-access-w8zj7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:17:37.324889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.324854 4959 scope.go:122] "RemoveContainer" containerID="9630128a3abba7ed84237e04425804d51c511706f4863ea82f2f1fbb8902f7f3" Oct 30 20:17:37.337083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.337062 4959 scope.go:122] "RemoveContainer" containerID="0617611d85df381290f39456b407683d0662d42d4d66a0d4abc9170461312cc2" Oct 30 20:17:37.613368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.613278 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb" Oct 30 20:17:37.685181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.685146 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:17:37.688581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.688550 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c8fc40-56648d78f5-99tjb"] Oct 30 20:17:37.770437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:37.770405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:38.001164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:38.001131 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" path="/var/lib/kubelet/pods/b925d0cb-a3cd-452e-9742-07d79a2928d0/volumes" Oct 30 20:17:39.113104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:39.113046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:39.580411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:39.580376 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:39.687724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:39.687692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.107277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:40.107237 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.507716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:40.507688 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:17:40.508187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:40.508119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:17:40.582478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:40.582448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:41.503992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:41.503949 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:43.051989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:43.051954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:43.804570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:43.804539 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.000558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:44.000481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.107193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:44.107118 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.367772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:44.367687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.479808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:44.479767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.584116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:44.584083 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:45.600314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:45.600281 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:17:45.600813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:45.600632 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:17:46.066436 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:17:46.066398 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:17:46.068896 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:46.068873 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.197323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:46.197273 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:48.041603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:48.041564 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:49.907232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:49.907195 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:50.625983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:50.625949 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:50.732473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:50.732439 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:50.990704 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:50.990668 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:51.837815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:51.837784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:52.535819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:52.535782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.002946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:54.002913 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.368825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:54.368741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:55.981883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:55.981836 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:57.713832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:57.713801 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:59.385593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:59.385558 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:59.489797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:17:59.489766 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:00.986557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:00.986514 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.879162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:01.879126 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.001133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:01.997744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.933142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:02.933110 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:04.562146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:04.562088 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:06.305029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:06.304983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:06.418122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:06.418089 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:06.531860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:06.531830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:08.337871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:08.337830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:10.373004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:10.372974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:12.107794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:12.107764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:12.213824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:12.213783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:14.134683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:14.134652 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:15.792920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:15.792888 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:16.066400 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:16.066316 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:18:17.460673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:17.460622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:17.944709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:17.944677 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:19.314224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:19.314178 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.106691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:21.106661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.529412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:21.529380 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:23.029274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:23.029237 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:25.016845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:25.016809 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.676222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.676193 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:18:26.676949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.676933 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" Oct 30 20:18:26.676998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.676951 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" Oct 30 20:18:26.677106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.677097 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b925d0cb-a3cd-452e-9742-07d79a2928d0" containerName="payload-pre-processing" Oct 30 20:18:26.682195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.682171 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.700364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.700335 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:18:26.707163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.707139 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-5c875d-serving-cert\"" Oct 30 20:18:26.758434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.758389 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.842483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.842438 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.842703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.842659 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.842803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.842785 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6l44\" (UniqueName: \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.842938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.842924 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.866808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.866775 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.949103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.944557 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6l44\" (UniqueName: \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.949103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.944700 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.949103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.944761 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.949103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.944939 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.950484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.950454 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.954004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.953858 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.954787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.954759 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.972341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.972307 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6l44\" (UniqueName: \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\") pod \"maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:26.974479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:26.974450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:27.001135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:27.001110 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:27.308820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:27.308747 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:18:27.312338 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:18:27.312309 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6799a8d5_19a3_4186_9433_99cfb35db04f.slice/crio-868ab939d6c9b38c0814b002f6bbcf093fa30111d2baff0d897e98ca0605694d WatchSource:0}: Error finding container 868ab939d6c9b38c0814b002f6bbcf093fa30111d2baff0d897e98ca0605694d: Status 404 returned error can't find the container with id 868ab939d6c9b38c0814b002f6bbcf093fa30111d2baff0d897e98ca0605694d Oct 30 20:18:28.009619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:28.009537 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:28.010165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:28.010138 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" event={"ID":"6799a8d5-19a3-4186-9433-99cfb35db04f","Type":"ContainerStarted","Data":"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935"} Oct 30 20:18:28.010283 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:28.010269 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" event={"ID":"6799a8d5-19a3-4186-9433-99cfb35db04f","Type":"ContainerStarted","Data":"868ab939d6c9b38c0814b002f6bbcf093fa30111d2baff0d897e98ca0605694d"} Oct 30 20:18:28.046517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:28.046343 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" podStartSLOduration=1.705315428 podStartE2EDuration="2.046323471s" totalImagesPullingTime="341.008043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:28.041541766 +0000 UTC m=+2632.170775691" watchObservedRunningTime="2026-10-30 20:18:28.046323471 +0000 UTC m=+2632.175557395" Oct 30 20:18:28.904455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:28.904419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:30.610383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:30.610341 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.958420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:31.958387 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:18:32.125554 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.125516 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:18:32.125809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.125778 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" podUID="758dba28-c590-419b-b34c-402cad2c77e8" containerName="maas-api" containerID="cri-o://3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca" gracePeriod=30 Oct 30 20:18:32.131268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.131235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:32.270892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.270860 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:32.431764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.431733 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:32.468927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.468898 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:18:32.545840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.545736 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\") pod \"758dba28-c590-419b-b34c-402cad2c77e8\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " Oct 30 20:18:32.546022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.545867 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\") pod \"758dba28-c590-419b-b34c-402cad2c77e8\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " Oct 30 20:18:32.546022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.545993 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\") pod \"758dba28-c590-419b-b34c-402cad2c77e8\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " Oct 30 20:18:32.546152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.546130 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\" (UniqueName: \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\") pod \"758dba28-c590-419b-b34c-402cad2c77e8\" (UID: \"758dba28-c590-419b-b34c-402cad2c77e8\") " Oct 30 20:18:32.546686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.546661 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca" pod "758dba28-c590-419b-b34c-402cad2c77e8" (UID: "758dba28-c590-419b-b34c-402cad2c77e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:32.549307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.549287 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs" pod "758dba28-c590-419b-b34c-402cad2c77e8" (UID: "758dba28-c590-419b-b34c-402cad2c77e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:32.549527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.549429 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls" pod "758dba28-c590-419b-b34c-402cad2c77e8" (UID: "758dba28-c590-419b-b34c-402cad2c77e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:32.551316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.551289 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn" pod "758dba28-c590-419b-b34c-402cad2c77e8" (UID: "758dba28-c590-419b-b34c-402cad2c77e8"). InnerVolumeSpecName "kube-api-access-fvxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:32.667762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.667725 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:32.667762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.667755 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvxmn\" (UniqueName: \"kubernetes.io/projected/758dba28-c590-419b-b34c-402cad2c77e8-kube-api-access-fvxmn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:32.667762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.667769 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/758dba28-c590-419b-b34c-402cad2c77e8-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:32.668099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.667781 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/758dba28-c590-419b-b34c-402cad2c77e8-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:32.944668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:32.944631 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:33.040349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.040313 4959 generic.go:356] "Generic (PLEG): container finished" podID="758dba28-c590-419b-b34c-402cad2c77e8" containerID="3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca" exitCode=0 Oct 30 20:18:33.040815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.040373 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" Oct 30 20:18:33.040815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.040403 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" event={"ID":"758dba28-c590-419b-b34c-402cad2c77e8","Type":"ContainerDied","Data":"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca"} Oct 30 20:18:33.040815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.040472 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp" event={"ID":"758dba28-c590-419b-b34c-402cad2c77e8","Type":"ContainerDied","Data":"be832e9b96db64f84b993e4324e0ad695a6048767bf31d9bfffe8d001d12054a"} Oct 30 20:18:33.040815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.040525 4959 scope.go:122] "RemoveContainer" containerID="3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca" Oct 30 20:18:33.071226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.071199 4959 scope.go:122] "RemoveContainer" containerID="3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca" Oct 30 20:18:33.071766 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:33.071738 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca\": container with ID starting with 3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca not found: ID does not exist" containerID="3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca" Oct 30 20:18:33.071865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.071779 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca"} err="failed to get container status \"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca\": rpc error: code = NotFound desc = could not find container \"3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca\": container with ID starting with 3d480fa626783f10ab8cb0bb62e6a9c880e970f792038b4c21f5283fb2be32ca not found: ID does not exist" Oct 30 20:18:33.126370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.126331 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:18:33.127868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.127817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:33.129745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.129728 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d3c508-7f659b9694-5gbwp"] Oct 30 20:18:33.235145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.235059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:33.601060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:33.600970 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:34.000236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:34.000199 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758dba28-c590-419b-b34c-402cad2c77e8" path="/var/lib/kubelet/pods/758dba28-c590-419b-b34c-402cad2c77e8/volumes" Oct 30 20:18:34.016070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:34.016041 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:18:35.048511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:35.048452 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:36.887566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:36.887405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:36.959585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:36.959548 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" containerID="cri-o://1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2" gracePeriod=25 Oct 30 20:18:38.242681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.242640 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:18:38.243212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.242897 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" podUID="7bf8063e-366b-4bf8-926f-22d93e41be8a" containerName="maas-api" containerID="cri-o://4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987" gracePeriod=30 Oct 30 20:18:38.327096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.327063 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.876211 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.871752 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:18:38.924007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.920016 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") pod \"7bf8063e-366b-4bf8-926f-22d93e41be8a\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " Oct 30 20:18:38.924007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.920143 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\") pod \"7bf8063e-366b-4bf8-926f-22d93e41be8a\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " Oct 30 20:18:38.924007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.920266 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\") pod \"7bf8063e-366b-4bf8-926f-22d93e41be8a\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " Oct 30 20:18:38.924007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.920335 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\" (UniqueName: \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\") pod \"7bf8063e-366b-4bf8-926f-22d93e41be8a\" (UID: \"7bf8063e-366b-4bf8-926f-22d93e41be8a\") " Oct 30 20:18:38.924007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.922085 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca" pod "7bf8063e-366b-4bf8-926f-22d93e41be8a" (UID: "7bf8063e-366b-4bf8-926f-22d93e41be8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:38.932805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.932766 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9" pod "7bf8063e-366b-4bf8-926f-22d93e41be8a" (UID: "7bf8063e-366b-4bf8-926f-22d93e41be8a"). InnerVolumeSpecName "kube-api-access-dk5p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:38.935444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.935188 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls" pod "7bf8063e-366b-4bf8-926f-22d93e41be8a" (UID: "7bf8063e-366b-4bf8-926f-22d93e41be8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:38.974942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.973052 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs" pod "7bf8063e-366b-4bf8-926f-22d93e41be8a" (UID: "7bf8063e-366b-4bf8-926f-22d93e41be8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:38.998447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:38.992404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.023632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.023592 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8063e-366b-4bf8-926f-22d93e41be8a-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:39.023838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.023629 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5p9\" (UniqueName: \"kubernetes.io/projected/7bf8063e-366b-4bf8-926f-22d93e41be8a-kube-api-access-dk5p9\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:39.023838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.023698 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:39.023838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.023712 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf8063e-366b-4bf8-926f-22d93e41be8a-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:39.085899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.085808 4959 generic.go:356] "Generic (PLEG): container finished" podID="7bf8063e-366b-4bf8-926f-22d93e41be8a" containerID="4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987" exitCode=0 Oct 30 20:18:39.085899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.085862 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" event={"ID":"7bf8063e-366b-4bf8-926f-22d93e41be8a","Type":"ContainerDied","Data":"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987"} Oct 30 20:18:39.085899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.085871 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" Oct 30 20:18:39.086213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.085909 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl" event={"ID":"7bf8063e-366b-4bf8-926f-22d93e41be8a","Type":"ContainerDied","Data":"b0311d96a6243c81d1d3be8c256d8ce97d1ea59cee4696198dad40abaacdcd15"} Oct 30 20:18:39.086213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.085935 4959 scope.go:122] "RemoveContainer" containerID="4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987" Oct 30 20:18:39.099323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.099302 4959 scope.go:122] "RemoveContainer" containerID="4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987" Oct 30 20:18:39.099651 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:39.099628 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987\": container with ID starting with 4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987 not found: ID does not exist" containerID="4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987" Oct 30 20:18:39.099747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.099672 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987"} err="failed to get container status \"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987\": rpc error: code = NotFound desc = could not find container \"4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987\": container with ID starting with 4e441d47d5c7e5ba9d2c235647992c8decb11d9f0929b0d39b09665051fbf987 not found: ID does not exist" Oct 30 20:18:39.206518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.205844 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.215625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.211017 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:18:39.232454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.232420 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a64be2-dc569f58-vsgzl"] Oct 30 20:18:39.361582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.356650 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:39.700519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.697658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.892927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:39.892889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:40.001419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:40.001330 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf8063e-366b-4bf8-926f-22d93e41be8a" path="/var/lib/kubelet/pods/7bf8063e-366b-4bf8-926f-22d93e41be8a/volumes" Oct 30 20:18:41.416359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:41.416322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.524188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:41.524156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.803840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:41.803754 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:42.499065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:42.499032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:44.625651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:44.625605 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.067192 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:46.067155 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:18:46.068934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.068914 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.179416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.179383 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.361839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.361611 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:18:46.362006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.361852 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" containerID="cri-o://58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a" gracePeriod=30 Oct 30 20:18:46.390421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.390391 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.671776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.671740 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.790995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:46.790965 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.099843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:49.099815 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.347181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:49.347146 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:50.672599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:50.672565 4959 patch_prober.go:29] interesting pod/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:18:50.673007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:50.672626 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:18:50.908753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:50.908720 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:51.640854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.640829 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:18:51.718824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.718797 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.718850 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.718883 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.718929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719045 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719113 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719115 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719212 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719242 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719261 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:51.719765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719326 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\") pod \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\" (UID: \"20477bc9-8bb0-451f-be9a-0d618d4cfb59\") " Oct 30 20:18:51.719765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.719542 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:51.720170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.720148 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.720245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.720177 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.720245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.720194 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.720447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.720424 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:51.720756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.720735 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:51.721657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.721636 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:51.721848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.721828 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "kube-api-access-6xh5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:51.722474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.722456 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:51.723763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.723740 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo" pod "20477bc9-8bb0-451f-be9a-0d618d4cfb59" (UID: "20477bc9-8bb0-451f-be9a-0d618d4cfb59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:51.821782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821750 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xh5s\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-kube-api-access-6xh5s\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.821782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821776 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.821782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821786 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.822041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821794 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.822041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821802 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20477bc9-8bb0-451f-be9a-0d618d4cfb59-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:51.822041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:51.821819 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20477bc9-8bb0-451f-be9a-0d618d4cfb59-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:52.215701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.215665 4959 generic.go:356] "Generic (PLEG): container finished" podID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerID="58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a" exitCode=0 Oct 30 20:18:52.215877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.215712 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" event={"ID":"20477bc9-8bb0-451f-be9a-0d618d4cfb59","Type":"ContainerDied","Data":"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a"} Oct 30 20:18:52.215877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.215753 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" event={"ID":"20477bc9-8bb0-451f-be9a-0d618d4cfb59","Type":"ContainerDied","Data":"c02af66ffa99639a06998558050fec19872756dc56230d29b93bc7baa4e1af89"} Oct 30 20:18:52.215877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.215775 4959 scope.go:122] "RemoveContainer" containerID="58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a" Oct 30 20:18:52.215877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.215710 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2" Oct 30 20:18:52.226610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.226588 4959 scope.go:122] "RemoveContainer" containerID="58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a" Oct 30 20:18:52.226885 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:52.226865 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a\": container with ID starting with 58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a not found: ID does not exist" containerID="58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a" Oct 30 20:18:52.226959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.226892 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a"} err="failed to get container status \"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a\": rpc error: code = NotFound desc = could not find container \"58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a\": container with ID starting with 58bc809b2f13df80452371f25728f291274f230f67d02e5d7dfa1e80ca3ac53a not found: ID does not exist" Oct 30 20:18:52.264956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.264899 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:18:52.270008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.269986 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a64be2-openshift-default-7f5575d8c9-pgcp2"] Oct 30 20:18:52.959020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:52.958989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:53.225020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:53.224921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:54.013435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:54.013396 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" path="/var/lib/kubelet/pods/20477bc9-8bb0-451f-be9a-0d618d4cfb59/volumes" Oct 30 20:18:55.506178 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:55.506147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:57.243992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.243968 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:18:57.248302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.248278 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:18:57.253660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.253627 4959 generic.go:356] "Generic (PLEG): container finished" podID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerID="1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2" exitCode=0 Oct 30 20:18:57.253794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.253669 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" Oct 30 20:18:57.253794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.253703 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" event={"ID":"2306e0ae-3564-437e-a17d-b5e15ba074f6","Type":"ContainerDied","Data":"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2"} Oct 30 20:18:57.253794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.253750 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7" event={"ID":"2306e0ae-3564-437e-a17d-b5e15ba074f6","Type":"ContainerDied","Data":"b8672c6bf49da234259a830421109efc27ae21b09f6bb630d806dd4ccdf12bc3"} Oct 30 20:18:57.253794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.253772 4959 scope.go:122] "RemoveContainer" containerID="1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2" Oct 30 20:18:57.264425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.264406 4959 scope.go:122] "RemoveContainer" containerID="1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2" Oct 30 20:18:57.264709 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:18:57.264686 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2\": container with ID starting with 1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2 not found: ID does not exist" containerID="1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2" Oct 30 20:18:57.264778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.264716 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2"} err="failed to get container status \"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2\": rpc error: code = NotFound desc = could not find container \"1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2\": container with ID starting with 1dece250240a29c424e60e27727ffc7c06bb116af69a87a80b097c5733de8cd2 not found: ID does not exist" Oct 30 20:18:57.279703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.279673 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:18:57.389820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.389787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:57.392975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.392948 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\") pod \"2306e0ae-3564-437e-a17d-b5e15ba074f6\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " Oct 30 20:18:57.393255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.393234 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\" (UniqueName: \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\") pod \"2306e0ae-3564-437e-a17d-b5e15ba074f6\" (UID: \"2306e0ae-3564-437e-a17d-b5e15ba074f6\") " Oct 30 20:18:57.393452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.393295 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume" pod "2306e0ae-3564-437e-a17d-b5e15ba074f6" (UID: "2306e0ae-3564-437e-a17d-b5e15ba074f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:57.393967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.393943 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2306e0ae-3564-437e-a17d-b5e15ba074f6-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:57.396976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.396947 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd" pod "2306e0ae-3564-437e-a17d-b5e15ba074f6" (UID: "2306e0ae-3564-437e-a17d-b5e15ba074f6"). InnerVolumeSpecName "kube-api-access-gq5vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:57.505893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.505800 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5vd\" (UniqueName: \"kubernetes.io/projected/2306e0ae-3564-437e-a17d-b5e15ba074f6-kube-api-access-gq5vd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:18:57.507210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.507118 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:57.604744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.604714 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:18:57.607869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.607839 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a64be2-68cc7b64cb-l2ng7"] Oct 30 20:18:57.786331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.786232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:57.998846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:57.998818 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" path="/var/lib/kubelet/pods/2306e0ae-3564-437e-a17d-b5e15ba074f6/volumes" Oct 30 20:18:58.049119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:58.049041 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:18:58.116951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:58.116913 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:18:58.234666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:58.234629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:58.345979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:58.345899 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:58.452927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:58.452879 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:59.862709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:18:59.862676 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:01.607261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:01.607223 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.249427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.249365 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" podUID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" containerName="payload-processing" containerID="cri-o://231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88" gracePeriod=25 Oct 30 20:19:02.280389 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.280358 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" containerID="cri-o://d5aec4d0d01cbc327881e9f10a518d43c9a6c12993be1c503cbc8549bfeb1790" gracePeriod=25 Oct 30 20:19:02.320160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.320131 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.559653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.559627 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:19:02.589110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.589082 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\") pod \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " Oct 30 20:19:02.589270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.589174 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\" (UniqueName: \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\") pod \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\" (UID: \"a0462ecf-61b4-4a94-9fa9-72b9ac5403da\") " Oct 30 20:19:02.589537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.589510 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume" pod "a0462ecf-61b4-4a94-9fa9-72b9ac5403da" (UID: "a0462ecf-61b4-4a94-9fa9-72b9ac5403da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:02.589657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.589644 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:02.591455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.591436 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t" pod "a0462ecf-61b4-4a94-9fa9-72b9ac5403da" (UID: "a0462ecf-61b4-4a94-9fa9-72b9ac5403da"). InnerVolumeSpecName "kube-api-access-8db8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:02.690622 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:02.690587 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8db8t\" (UniqueName: \"kubernetes.io/projected/a0462ecf-61b4-4a94-9fa9-72b9ac5403da-kube-api-access-8db8t\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:03.049475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.049415 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" podUID="3a974811-ce19-4073-8599-89d75aa07947" containerName="payload-processing" containerID="cri-o://f7c2ab2f38db94b97e1e91904f79048f11c90856b86c4aae190b6185af5ced5f" gracePeriod=25 Oct 30 20:19:03.117442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.117403 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" podUID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" containerName="payload-pre-processing" containerID="cri-o://76649f05ced98ab06b2134dd7c685539cfc0ecde0a51851d90fdf2cb775afc1f" gracePeriod=25 Oct 30 20:19:03.308225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.308148 4959 generic.go:356] "Generic (PLEG): container finished" podID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" containerID="231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88" exitCode=0 Oct 30 20:19:03.308363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.308290 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" event={"ID":"a0462ecf-61b4-4a94-9fa9-72b9ac5403da","Type":"ContainerDied","Data":"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88"} Oct 30 20:19:03.308363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.308344 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" event={"ID":"a0462ecf-61b4-4a94-9fa9-72b9ac5403da","Type":"ContainerDied","Data":"ad630a4b23ad0a727bd96ff0eceb2ae42268121c43a97b95b2f9fb25c0401ebe"} Oct 30 20:19:03.308520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.308372 4959 scope.go:122] "RemoveContainer" containerID="231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88" Oct 30 20:19:03.308659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.308639 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt" Oct 30 20:19:03.313145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.312525 4959 generic.go:356] "Generic (PLEG): container finished" podID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" containerID="76649f05ced98ab06b2134dd7c685539cfc0ecde0a51851d90fdf2cb775afc1f" exitCode=0 Oct 30 20:19:03.313145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.312734 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" event={"ID":"5d7b2a8c-79dd-4659-a2a6-faf932938ae0","Type":"ContainerDied","Data":"76649f05ced98ab06b2134dd7c685539cfc0ecde0a51851d90fdf2cb775afc1f"} Oct 30 20:19:03.316591 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.316555 4959 generic.go:356] "Generic (PLEG): container finished" podID="3a974811-ce19-4073-8599-89d75aa07947" containerID="f7c2ab2f38db94b97e1e91904f79048f11c90856b86c4aae190b6185af5ced5f" exitCode=0 Oct 30 20:19:03.316716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.316665 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" event={"ID":"3a974811-ce19-4073-8599-89d75aa07947","Type":"ContainerDied","Data":"f7c2ab2f38db94b97e1e91904f79048f11c90856b86c4aae190b6185af5ced5f"} Oct 30 20:19:03.327626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.326603 4959 scope.go:122] "RemoveContainer" containerID="231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88" Oct 30 20:19:03.327626 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:19:03.326912 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88\": container with ID starting with 231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88 not found: ID does not exist" containerID="231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88" Oct 30 20:19:03.327626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.326935 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88"} err="failed to get container status \"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88\": rpc error: code = NotFound desc = could not find container \"231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88\": container with ID starting with 231d0261a3283cee68698982825667aef481294d7b0822d7d0226690756e7e88 not found: ID does not exist" Oct 30 20:19:03.374346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.374016 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:19:03.374346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.374070 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4cfcee-899db95f8-8sxrt"] Oct 30 20:19:03.405287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.405244 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:19:03.408646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.408615 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:03.500467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.500426 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\" (UniqueName: \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\") pod \"3a974811-ce19-4073-8599-89d75aa07947\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " Oct 30 20:19:03.500991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.500963 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\") pod \"3a974811-ce19-4073-8599-89d75aa07947\" (UID: \"3a974811-ce19-4073-8599-89d75aa07947\") " Oct 30 20:19:03.502143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.502108 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume" pod "3a974811-ce19-4073-8599-89d75aa07947" (UID: "3a974811-ce19-4073-8599-89d75aa07947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:03.502371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.502348 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:19:03.504532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.504467 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z" pod "3a974811-ce19-4073-8599-89d75aa07947" (UID: "3a974811-ce19-4073-8599-89d75aa07947"). InnerVolumeSpecName "kube-api-access-cvg7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:03.602663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.602582 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\") pod \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " Oct 30 20:19:03.602663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.602630 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\" (UniqueName: \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\") pod \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\" (UID: \"5d7b2a8c-79dd-4659-a2a6-faf932938ae0\") " Oct 30 20:19:03.602971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.602948 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume" pod "5d7b2a8c-79dd-4659-a2a6-faf932938ae0" (UID: "5d7b2a8c-79dd-4659-a2a6-faf932938ae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:03.603169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.603156 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a974811-ce19-4073-8599-89d75aa07947-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:03.603209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.603173 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvg7z\" (UniqueName: \"kubernetes.io/projected/3a974811-ce19-4073-8599-89d75aa07947-kube-api-access-cvg7z\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:03.603209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.603181 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:03.604941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.604917 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4" pod "5d7b2a8c-79dd-4659-a2a6-faf932938ae0" (UID: "5d7b2a8c-79dd-4659-a2a6-faf932938ae0"). InnerVolumeSpecName "kube-api-access-v5tj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:03.704255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.704223 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5tj4\" (UniqueName: \"kubernetes.io/projected/5d7b2a8c-79dd-4659-a2a6-faf932938ae0-kube-api-access-v5tj4\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:03.998972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:03.998940 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" path="/var/lib/kubelet/pods/a0462ecf-61b4-4a94-9fa9-72b9ac5403da/volumes" Oct 30 20:19:04.327698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.327669 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" Oct 30 20:19:04.327905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.327656 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84" event={"ID":"5d7b2a8c-79dd-4659-a2a6-faf932938ae0","Type":"ContainerDied","Data":"a3ee7e2a6a166d07000a2b877f55332fc133aa4da1b88cc5dcd1e2c0ad483f5d"} Oct 30 20:19:04.327905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.327816 4959 scope.go:122] "RemoveContainer" containerID="76649f05ced98ab06b2134dd7c685539cfc0ecde0a51851d90fdf2cb775afc1f" Oct 30 20:19:04.329333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.329307 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" event={"ID":"3a974811-ce19-4073-8599-89d75aa07947","Type":"ContainerDied","Data":"aff1f2582bd2e69443ea519ee54bbf759866e3bea463e1491afa7d4eed0c6694"} Oct 30 20:19:04.329333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.329329 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw" Oct 30 20:19:04.340672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.340654 4959 scope.go:122] "RemoveContainer" containerID="f7c2ab2f38db94b97e1e91904f79048f11c90856b86c4aae190b6185af5ced5f" Oct 30 20:19:04.377838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.377812 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:19:04.388577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.388554 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-188d45-5c8d8dcb68-trthw"] Oct 30 20:19:04.412583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.412558 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:19:04.416317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:04.416292 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-188d45-56d8b4b8dc-r7b84"] Oct 30 20:19:05.109012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:05.108982 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:05.998506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:05.998467 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a974811-ce19-4073-8599-89d75aa07947" path="/var/lib/kubelet/pods/3a974811-ce19-4073-8599-89d75aa07947/volumes" Oct 30 20:19:05.998974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:05.998958 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" path="/var/lib/kubelet/pods/5d7b2a8c-79dd-4659-a2a6-faf932938ae0/volumes" Oct 30 20:19:06.815608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:06.815576 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:08.009618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:08.009589 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:19:08.605138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:08.605103 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:10.230810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:10.230775 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:11.884094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:11.884060 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:13.679365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:13.679332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:15.473748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:15.473717 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:16.066754 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:19:16.066719 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:19:16.430369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:16.430335 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:17.161564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:17.161534 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:18.009026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.008993 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:19:18.907507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.907460 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:19:18.908132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908118 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7bf8063e-366b-4bf8-926f-22d93e41be8a" containerName="maas-api" Oct 30 20:19:18.908184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908134 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf8063e-366b-4bf8-926f-22d93e41be8a" containerName="maas-api" Oct 30 20:19:18.908184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908156 4959 cpu_manager.go:414] "containerMap: removing container" podUID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" containerName="payload-pre-processing" Oct 30 20:19:18.908184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908161 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" containerName="payload-pre-processing" Oct 30 20:19:18.908184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908172 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" containerName="payload-processing" Oct 30 20:19:18.908184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908177 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908193 4959 cpu_manager.go:414] "containerMap: removing container" podUID="758dba28-c590-419b-b34c-402cad2c77e8" containerName="maas-api" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908198 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758dba28-c590-419b-b34c-402cad2c77e8" containerName="maas-api" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908221 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908225 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908238 4959 cpu_manager.go:414] "containerMap: removing container" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908242 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908249 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3a974811-ce19-4073-8599-89d75aa07947" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908254 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a974811-ce19-4073-8599-89d75aa07947" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908341 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2306e0ae-3564-437e-a17d-b5e15ba074f6" containerName="payload-processing" Oct 30 20:19:18.908350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908352 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="20477bc9-8bb0-451f-be9a-0d618d4cfb59" containerName="istio-proxy" Oct 30 20:19:18.908685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908357 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a974811-ce19-4073-8599-89d75aa07947" containerName="payload-processing" Oct 30 20:19:18.908685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908364 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf8063e-366b-4bf8-926f-22d93e41be8a" containerName="maas-api" Oct 30 20:19:18.908685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908371 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7b2a8c-79dd-4659-a2a6-faf932938ae0" containerName="payload-pre-processing" Oct 30 20:19:18.908685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908377 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0462ecf-61b4-4a94-9fa9-72b9ac5403da" containerName="payload-processing" Oct 30 20:19:18.908685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.908385 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="758dba28-c590-419b-b34c-402cad2c77e8" containerName="maas-api" Oct 30 20:19:18.913446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.913421 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:18.917416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.917381 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:18.941668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.941636 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"kube-root-ca.crt\"" Oct 30 20:19:18.941912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.941805 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"auth-test-model-77ac5a-kserve-self-signed-certs\"" Oct 30 20:19:18.942013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.941856 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"openshift-service-ca.crt\"" Oct 30 20:19:18.942081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.942046 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-77ac5a\"/\"default-dockercfg-72phh\"" Oct 30 20:19:18.947525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:18.947460 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:19:19.006746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.006716 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxwz\" (UniqueName: \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.006934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.006764 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.006934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.006855 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.006934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.006894 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.006934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.006913 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.007077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.007024 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.032918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.032868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:19.108772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.108738 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.108953 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.108923 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxwz\" (UniqueName: \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.108959 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.108985 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.109014 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.109029 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.109399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.109436 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.109557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.109460 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.111286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.111257 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.111600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.111582 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.127321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.127296 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxwz\" (UniqueName: \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\") pod \"auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.230610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.230522 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:19.553409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:19.553328 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:19:19.556898 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:19.556871 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefd6afec_8257_468f_a4e9_6bbcd2061063.slice/crio-65a5dbf5898517b20b4a3e4fe4c67e88a90bf68b1a7b3743b4faedb76330583f WatchSource:0}: Error finding container 65a5dbf5898517b20b4a3e4fe4c67e88a90bf68b1a7b3743b4faedb76330583f: Status 404 returned error can't find the container with id 65a5dbf5898517b20b4a3e4fe4c67e88a90bf68b1a7b3743b4faedb76330583f Oct 30 20:19:20.441981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:20.441946 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" event={"ID":"efd6afec-8257-468f-a4e9-6bbcd2061063","Type":"ContainerStarted","Data":"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe"} Oct 30 20:19:20.441981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:20.441987 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" event={"ID":"efd6afec-8257-468f-a4e9-6bbcd2061063","Type":"ContainerStarted","Data":"65a5dbf5898517b20b4a3e4fe4c67e88a90bf68b1a7b3743b4faedb76330583f"} Oct 30 20:19:20.482106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:20.482051 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" podStartSLOduration=2.482034947 podStartE2EDuration="2.482034947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:20.479159347 +0000 UTC m=+2684.608393281" watchObservedRunningTime="2026-10-30 20:19:20.482034947 +0000 UTC m=+2684.611268868" Oct 30 20:19:20.762337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:20.762036 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:21.231653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:21.231603 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:22.392356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.392312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:22.461126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.461072 4959 generic.go:356] "Generic (PLEG): container finished" podID="13b2f127-2014-402e-8c65-bf0e85972e88" containerID="d5aec4d0d01cbc327881e9f10a518d43c9a6c12993be1c503cbc8549bfeb1790" exitCode=0 Oct 30 20:19:22.461392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.461364 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" event={"ID":"13b2f127-2014-402e-8c65-bf0e85972e88","Type":"ContainerDied","Data":"d5aec4d0d01cbc327881e9f10a518d43c9a6c12993be1c503cbc8549bfeb1790"} Oct 30 20:19:22.577004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.576978 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:19:22.653467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.653432 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\") pod \"13b2f127-2014-402e-8c65-bf0e85972e88\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " Oct 30 20:19:22.653698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.653680 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\" (UniqueName: \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\") pod \"13b2f127-2014-402e-8c65-bf0e85972e88\" (UID: \"13b2f127-2014-402e-8c65-bf0e85972e88\") " Oct 30 20:19:22.653918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.653887 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume" pod "13b2f127-2014-402e-8c65-bf0e85972e88" (UID: "13b2f127-2014-402e-8c65-bf0e85972e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:22.654632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.654551 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b2f127-2014-402e-8c65-bf0e85972e88-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:22.656109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.656086 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t" pod "13b2f127-2014-402e-8c65-bf0e85972e88" (UID: "13b2f127-2014-402e-8c65-bf0e85972e88"). InnerVolumeSpecName "kube-api-access-tzl8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:22.755899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:22.755864 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl8t\" (UniqueName: \"kubernetes.io/projected/13b2f127-2014-402e-8c65-bf0e85972e88-kube-api-access-tzl8t\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:23.470432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.470393 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" Oct 30 20:19:23.470918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.470430 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w" event={"ID":"13b2f127-2014-402e-8c65-bf0e85972e88","Type":"ContainerDied","Data":"1560c9e6c571c1a7c236463ff81394bf3492f4159735a0cb42cbb3bf0584d6bd"} Oct 30 20:19:23.470918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.470504 4959 scope.go:122] "RemoveContainer" containerID="d5aec4d0d01cbc327881e9f10a518d43c9a6c12993be1c503cbc8549bfeb1790" Oct 30 20:19:23.519304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.519272 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:19:23.529819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.529787 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4cfcee-574dc78879-cnw9w"] Oct 30 20:19:23.998064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:23.998022 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" path="/var/lib/kubelet/pods/13b2f127-2014-402e-8c65-bf0e85972e88/volumes" Oct 30 20:19:24.089753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:24.089716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:25.825461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:25.825409 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:27.228340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:27.228311 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:27.917630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:27.917590 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.028707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:28.028671 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.141056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:28.141020 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.574078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:28.574029 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:29.231185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:29.231146 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:29.244909 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:29.244882 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:29.525515 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:29.525416 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:19:30.056697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:30.056662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:31.682039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:31.681998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:31.856337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:31.856285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.472964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:32.472930 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:33.062154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:33.062120 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:35.007903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.007869 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:35.239514 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.239458 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:35.344062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.343886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:35.568466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.568430 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:19:35.579603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.577802 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" containerID="cri-o://b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35" gracePeriod=30 Oct 30 20:19:35.602278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:35.602193 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:36.424914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:36.424887 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:19:36.431748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:36.431727 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:19:36.579180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:36.579146 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:37.069020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:37.068988 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:37.205077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:37.205040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:37.428665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:37.428636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:38.644980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:38.644937 4959 patch_prober.go:29] interesting pod/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:19:38.645445 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:38.645006 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:19:38.977020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:38.976911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:40.503033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:40.502952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:40.616425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:40.616390 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:40.822738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:40.822701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:40.853840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:40.853815 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:19:41.023630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023598 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.023855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023645 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.023855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023724 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.023855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023758 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.023855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023796 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.024097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023943 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.024097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023943 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:41.024097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.023996 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.024097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024073 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.024331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024111 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\") pod \"3a6a0f85-0b17-4393-a20c-00da2257498a\" (UID: \"3a6a0f85-0b17-4393-a20c-00da2257498a\") " Oct 30 20:19:41.024331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024161 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:41.024331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024320 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:41.024535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024329 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:19:41.024641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.024615 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:41.025075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.025052 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.025175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.025080 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6a0f85-0b17-4393-a20c-00da2257498a-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.025175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.025096 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.025175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.025112 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.025175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.025126 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.026481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.026454 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:19:41.026640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.026510 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:19:41.026640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.026558 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:41.026854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.026832 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf" pod "3a6a0f85-0b17-4393-a20c-00da2257498a" (UID: "3a6a0f85-0b17-4393-a20c-00da2257498a"). InnerVolumeSpecName "kube-api-access-zgmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.127481 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.127531 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.127544 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6a0f85-0b17-4393-a20c-00da2257498a-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.127556 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgmsf\" (UniqueName: \"kubernetes.io/projected/3a6a0f85-0b17-4393-a20c-00da2257498a-kube-api-access-zgmsf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.128221 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129038 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129054 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129068 4959 cpu_manager.go:414] "containerMap: removing container" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129074 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129212 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:41.130200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.129229 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b2f127-2014-402e-8c65-bf0e85972e88" containerName="payload-pre-processing" Oct 30 20:19:41.133673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.133648 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.159806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.159777 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:19:41.167110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.167086 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"default-dockercfg-92gh8\"" Oct 30 20:19:41.167235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.167185 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"auth-test-model-5314c2-kserve-self-signed-certs\"" Oct 30 20:19:41.227864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.227831 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"openshift-service-ca.crt\"" Oct 30 20:19:41.236838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.227965 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5314c2\"/\"kube-root-ca.crt\"" Oct 30 20:19:41.242235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.242099 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:41.335567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.335486 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.335780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.335684 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xw9\" (UniqueName: \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.335780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.335719 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.335899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.335856 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.335956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.335905 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.336211 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.336033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437261 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437363 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437469 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xw9\" (UniqueName: \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437538 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437585 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437616 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.437986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.437960 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.438088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.438001 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.438088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.438040 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.439863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.439835 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.440107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.440091 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.456394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.456367 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xw9\" (UniqueName: \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\") pod \"auth-test-model-5314c2-kserve-996f5b5df-6xz76\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:41.608212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.608180 4959 generic.go:356] "Generic (PLEG): container finished" podID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerID="b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35" exitCode=0 Oct 30 20:19:41.608627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.608242 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" Oct 30 20:19:41.608627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.608270 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerDied","Data":"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35"} Oct 30 20:19:41.608627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.608339 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t" event={"ID":"3a6a0f85-0b17-4393-a20c-00da2257498a","Type":"ContainerDied","Data":"175fb88687b369862bcef2f33450c976efb3de9c8537cedee9b691218fb5cc86"} Oct 30 20:19:41.608627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.608371 4959 scope.go:122] "RemoveContainer" containerID="b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35" Oct 30 20:19:41.620423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.620402 4959 scope.go:122] "RemoveContainer" containerID="67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1" Oct 30 20:19:41.630693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.630671 4959 scope.go:122] "RemoveContainer" containerID="b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35" Oct 30 20:19:41.630985 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:19:41.630964 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35\": container with ID starting with b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35 not found: ID does not exist" containerID="b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35" Oct 30 20:19:41.631058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.630994 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35"} err="failed to get container status \"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35\": rpc error: code = NotFound desc = could not find container \"b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35\": container with ID starting with b7186153096e6c759447a64e621972e075f57a08ca0ba7bcb4ab2ef73cf45a35 not found: ID does not exist" Oct 30 20:19:41.631058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.631017 4959 scope.go:122] "RemoveContainer" containerID="67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1" Oct 30 20:19:41.631262 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:19:41.631244 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1\": container with ID starting with 67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1 not found: ID does not exist" containerID="67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1" Oct 30 20:19:41.631308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.631271 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1"} err="failed to get container status \"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1\": rpc error: code = NotFound desc = could not find container \"67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1\": container with ID starting with 67a02d9fd0346219190423b82d0f870a6ab2f6e740cffa73af661693abd1c8b1 not found: ID does not exist" Oct 30 20:19:41.674124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.674092 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:19:41.681834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.681807 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4cfcee-openshift-default-5b96c4b96f-p4b9t"] Oct 30 20:19:41.752012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:41.751922 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:42.000073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.000040 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" path="/var/lib/kubelet/pods/3a6a0f85-0b17-4393-a20c-00da2257498a/volumes" Oct 30 20:19:42.061015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.060990 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:19:42.063405 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:42.063354 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd99cfa0b_a981_49ec_a851_333206b31117.slice/crio-2b26dbb5b327f9d6797b49a62b681001eb28da462e60cadf9c72a793e298d163 WatchSource:0}: Error finding container 2b26dbb5b327f9d6797b49a62b681001eb28da462e60cadf9c72a793e298d163: Status 404 returned error can't find the container with id 2b26dbb5b327f9d6797b49a62b681001eb28da462e60cadf9c72a793e298d163 Oct 30 20:19:42.622818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.622779 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" event={"ID":"d99cfa0b-a981-49ec-a851-333206b31117","Type":"ContainerStarted","Data":"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511"} Oct 30 20:19:42.622818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.622833 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" event={"ID":"d99cfa0b-a981-49ec-a851-333206b31117","Type":"ContainerStarted","Data":"2b26dbb5b327f9d6797b49a62b681001eb28da462e60cadf9c72a793e298d163"} Oct 30 20:19:42.662359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.662297 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" podStartSLOduration=1.6622834069999999 podStartE2EDuration="1.662283407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:42.661147271 +0000 UTC m=+2706.790381191" watchObservedRunningTime="2026-10-30 20:19:42.662283407 +0000 UTC m=+2706.791517327" Oct 30 20:19:42.752581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.752467 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:42.926869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:42.926829 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:43.072885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.072841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:43.180600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.180508 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:43.252632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.252602 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:19:43.253657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.253637 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:43.253716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.253661 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:43.253840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.253827 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a0f85-0b17-4393-a20c-00da2257498a" containerName="istio-proxy" Oct 30 20:19:43.257475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.257455 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.271377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.271347 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6175d5-dockercfg-zjfln\"" Oct 30 20:19:43.271377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.271353 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6175d5\"" Oct 30 20:19:43.274763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.274739 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:19:43.397204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.386738 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqsk\" (UniqueName: \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.397204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.386841 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.397204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.387361 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:19:43.397595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.397396 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.398458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.398433 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:19:43.488060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.487970 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.488060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.488010 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqsk\" (UniqueName: \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.488271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.488100 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwd5\" (UniqueName: \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.488271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.488150 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.488839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.488821 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.506711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.506680 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqsk\" (UniqueName: \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\") pod \"payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.571803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.571763 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:43.589327 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.589301 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwd5\" (UniqueName: \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.589528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.589512 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.590183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.590159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.608031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.608005 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwd5\" (UniqueName: \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\") pod \"payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.712072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.712028 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:43.876345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:43.876315 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:19:43.880008 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:43.879978 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d80d4c_8615_4f5d_ac4e_5db9c3f4c645.slice/crio-a57bfc4c9464a67a3f4d1b782f7ae1029af08b9eea8e646d2ede8e1662371141 WatchSource:0}: Error finding container a57bfc4c9464a67a3f4d1b782f7ae1029af08b9eea8e646d2ede8e1662371141: Status 404 returned error can't find the container with id a57bfc4c9464a67a3f4d1b782f7ae1029af08b9eea8e646d2ede8e1662371141 Oct 30 20:19:44.015700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.015654 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:19:44.019472 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:44.019443 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3db2d5_0f91_4a7f_a0ca_1c0dfa77e758.slice/crio-f7b838a8dea5e2f058d9183790ff4e9659cca8004fd20df4292f8affe5a96eab WatchSource:0}: Error finding container f7b838a8dea5e2f058d9183790ff4e9659cca8004fd20df4292f8affe5a96eab: Status 404 returned error can't find the container with id f7b838a8dea5e2f058d9183790ff4e9659cca8004fd20df4292f8affe5a96eab Oct 30 20:19:44.643568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.643531 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" event={"ID":"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758","Type":"ContainerStarted","Data":"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46"} Oct 30 20:19:44.643796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.643598 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" event={"ID":"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758","Type":"ContainerStarted","Data":"f7b838a8dea5e2f058d9183790ff4e9659cca8004fd20df4292f8affe5a96eab"} Oct 30 20:19:44.643796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.643632 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:44.645235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.645206 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" event={"ID":"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645","Type":"ContainerStarted","Data":"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9"} Oct 30 20:19:44.645367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.645254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" event={"ID":"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645","Type":"ContainerStarted","Data":"a57bfc4c9464a67a3f4d1b782f7ae1029af08b9eea8e646d2ede8e1662371141"} Oct 30 20:19:44.645426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.645393 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:44.682091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.682030 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" podStartSLOduration=1.6820148700000002 podStartE2EDuration="1.68201487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:44.680989221 +0000 UTC m=+2708.810223143" watchObservedRunningTime="2026-10-30 20:19:44.68201487 +0000 UTC m=+2708.811248785" Oct 30 20:19:44.725141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.725082 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" podStartSLOduration=1.72506745 podStartE2EDuration="1.72506745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:44.72154115 +0000 UTC m=+2708.850775073" watchObservedRunningTime="2026-10-30 20:19:44.72506745 +0000 UTC m=+2708.854301367" Oct 30 20:19:44.759342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:44.759311 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:46.067219 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:19:46.067174 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:19:46.660819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:46.660779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:48.425950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:48.425913 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:50.220933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:50.220899 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:50.858801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:50.858770 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:51.292776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.292743 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:19:51.298155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.298129 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.312092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.312068 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-30e92c-openshift-default-dockercfg-9lqrh\"" Oct 30 20:19:51.328201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.328167 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:19:51.351976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.351944 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:51.387041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387008 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387081 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387147 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387186 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf8fq\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387216 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387288 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387322 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387462 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.387573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.387560 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.471541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.469071 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:51.489025 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.488988 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489062 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489140 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489218 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489264 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489289 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf8fq\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489325 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489378 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.489403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.489405 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.491600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.490048 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.491600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.490327 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.492670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.492628 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.493478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.493452 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.498469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.496399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.506698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.501457 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.514676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.514532 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.519075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.519045 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.523582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.523553 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf8fq\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\") pod \"e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.618800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.618710 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:51.752379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.752334 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:51.764965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.764936 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:51.945907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.945811 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:19:51.948914 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:51.948878 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod618a7097_47ef_4d5b_8474_4353dface889.slice/crio-eaf4853141ea95cfd0c429bf69f26265c2fb8d29ac5940ff9448e755ecd9cac0 WatchSource:0}: Error finding container eaf4853141ea95cfd0c429bf69f26265c2fb8d29ac5940ff9448e755ecd9cac0: Status 404 returned error can't find the container with id eaf4853141ea95cfd0c429bf69f26265c2fb8d29ac5940ff9448e755ecd9cac0 Oct 30 20:19:51.951902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.951869 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:19:51.951978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.951946 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:19:51.952014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:51.951977 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:19:52.112555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:52.112516 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:52.718826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:52.718789 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerStarted","Data":"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0"} Oct 30 20:19:52.719330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:52.718850 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerStarted","Data":"eaf4853141ea95cfd0c429bf69f26265c2fb8d29ac5940ff9448e755ecd9cac0"} Oct 30 20:19:52.730205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:52.730182 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:19:52.773329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:52.773276 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podStartSLOduration=1.77326166 podStartE2EDuration="1.77326166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:52.772349167 +0000 UTC m=+2716.901583088" watchObservedRunningTime="2026-10-30 20:19:52.77326166 +0000 UTC m=+2716.902495580" Oct 30 20:19:53.619788 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:53.619752 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:53.625909 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:53.625883 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:53.725937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:53.725901 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:53.726762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:53.726741 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:19:53.899130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:53.899090 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:54.844229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:54.844198 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:55.342356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:55.342322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:55.511385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:55.511343 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:55.629168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:55.629081 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:55.655990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:55.655959 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:19:55.656138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:55.656044 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:19:56.291149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:56.291097 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:57.137931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.137895 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:57.237072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.237044 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:19:57.242646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.242621 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.244858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.244833 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:57.269564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.269533 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30e92c\"" Oct 30 20:19:57.269795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.269774 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30e92c-dockercfg-85fkb\"" Oct 30 20:19:57.274444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.274420 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:19:57.374596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.374568 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.374962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.374623 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjhm\" (UniqueName: \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.483240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.478399 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjhm\" (UniqueName: \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.494747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.484381 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.494747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.485464 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.508136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.508095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:57.523183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.516367 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjhm\" (UniqueName: \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\") pod \"payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.559370 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.559335 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:57.875248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:57.875155 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:19:57.879347 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:19:57.879319 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20e90aa_2296_4980_bd51_eb62f71444c7.slice/crio-f20a8d33f18c58d7c1409112f7fde1b99a1765701f2c44a64390b9d55ba66750 WatchSource:0}: Error finding container f20a8d33f18c58d7c1409112f7fde1b99a1765701f2c44a64390b9d55ba66750: Status 404 returned error can't find the container with id f20a8d33f18c58d7c1409112f7fde1b99a1765701f2c44a64390b9d55ba66750 Oct 30 20:19:58.699622 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:58.699578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:58.771577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:58.771538 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" event={"ID":"c20e90aa-2296-4980-bd51-eb62f71444c7","Type":"ContainerStarted","Data":"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2"} Oct 30 20:19:58.771784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:58.771607 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" event={"ID":"c20e90aa-2296-4980-bd51-eb62f71444c7","Type":"ContainerStarted","Data":"f20a8d33f18c58d7c1409112f7fde1b99a1765701f2c44a64390b9d55ba66750"} Oct 30 20:19:58.771784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:58.771640 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:19:58.811699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:19:58.811632 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" podStartSLOduration=1.811613387 podStartE2EDuration="1.811613387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:58.811295587 +0000 UTC m=+2722.940529510" watchObservedRunningTime="2026-10-30 20:19:58.811613387 +0000 UTC m=+2722.940847305" Oct 30 20:20:00.627156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:00.627123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:02.385413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:02.385368 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.060443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:04.060412 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.174044 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:04.173989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:05.071589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:05.071544 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:05.816532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:05.815246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:07.840950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:07.840916 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:09.189133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.189100 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7053-44d415"] Oct 30 20:20:09.194794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.194777 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:09.209075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.209049 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-44d415"] Oct 30 20:20:09.216653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.216626 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-brtz2\"" Oct 30 20:20:09.243079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.243040 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmp29\" (UniqueName: \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\") pod \"test-curl-7053-44d415\" (UID: \"4cf9fdac-5fdf-47a7-aff5-476258034291\") " pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:09.344610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.344575 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmp29\" (UniqueName: \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\") pod \"test-curl-7053-44d415\" (UID: \"4cf9fdac-5fdf-47a7-aff5-476258034291\") " pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:09.364245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.364219 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmp29\" (UniqueName: \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\") pod \"test-curl-7053-44d415\" (UID: \"4cf9fdac-5fdf-47a7-aff5-476258034291\") " pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:09.507990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.507904 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:09.698222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.698185 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:09.782360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.782280 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:20:09.820107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.820077 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7053-44d415"] Oct 30 20:20:09.847874 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:20:09.843845 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cf9fdac_5fdf_47a7_aff5_476258034291.slice/crio-82335b250f3cd1aac70df2075c9cb9348d8fddc6861de2616f3506289e22af65 WatchSource:0}: Error finding container 82335b250f3cd1aac70df2075c9cb9348d8fddc6861de2616f3506289e22af65: Status 404 returned error can't find the container with id 82335b250f3cd1aac70df2075c9cb9348d8fddc6861de2616f3506289e22af65 Oct 30 20:20:09.866276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:09.866239 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-44d415" event={"ID":"4cf9fdac-5fdf-47a7-aff5-476258034291","Type":"ContainerStarted","Data":"82335b250f3cd1aac70df2075c9cb9348d8fddc6861de2616f3506289e22af65"} Oct 30 20:20:10.874677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:10.874633 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7053-44d415" event={"ID":"4cf9fdac-5fdf-47a7-aff5-476258034291","Type":"ContainerStarted","Data":"023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47"} Oct 30 20:20:10.912953 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:10.912890 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7053-44d415" podStartSLOduration=1.9128774750000002 podStartE2EDuration="1.912877475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:10.909715358 +0000 UTC m=+2735.038949281" watchObservedRunningTime="2026-10-30 20:20:10.912877475 +0000 UTC m=+2735.042111396" Oct 30 20:20:11.421899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:11.421862 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:11.783742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:11.783651 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7053-44d415"] Oct 30 20:20:11.786862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:11.786832 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7053-44d415"] Oct 30 20:20:12.890664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:12.890604 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7053-44d415" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" containerName="test-curl-7053-44d415" containerID="cri-o://023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47" gracePeriod=2 Oct 30 20:20:13.656190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:13.656156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:15.104691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.104666 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:15.228722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.228645 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\" (UniqueName: \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\") pod \"4cf9fdac-5fdf-47a7-aff5-476258034291\" (UID: \"4cf9fdac-5fdf-47a7-aff5-476258034291\") " Oct 30 20:20:15.230941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.230908 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29" pod "4cf9fdac-5fdf-47a7-aff5-476258034291" (UID: "4cf9fdac-5fdf-47a7-aff5-476258034291"). InnerVolumeSpecName "kube-api-access-bmp29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:15.330802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.330767 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmp29\" (UniqueName: \"kubernetes.io/projected/4cf9fdac-5fdf-47a7-aff5-476258034291-kube-api-access-bmp29\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:20:15.410283 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.410240 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:15.913967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.913933 4959 generic.go:356] "Generic (PLEG): container finished" podID="4cf9fdac-5fdf-47a7-aff5-476258034291" containerID="023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47" exitCode=137 Oct 30 20:20:15.914150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.913980 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:15.914150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.914036 4959 scope.go:122] "RemoveContainer" containerID="023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47" Oct 30 20:20:15.924488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.924468 4959 scope.go:122] "RemoveContainer" containerID="023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47" Oct 30 20:20:15.924777 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:20:15.924759 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47\": container with ID starting with 023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47 not found: ID does not exist" containerID="023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47" Oct 30 20:20:15.924829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.924784 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47"} err="failed to get container status \"023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47\": rpc error: code = NotFound desc = could not find container \"023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47\": container with ID starting with 023e7352759f65ca8c375b8eb4c597e63fec8ebf305e4fcbdd862e8811489b47 not found: ID does not exist" Oct 30 20:20:15.935205 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:20:15.935181 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-44d415\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:15.999095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:15.999065 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" path="/var/lib/kubelet/pods/4cf9fdac-5fdf-47a7-aff5-476258034291/volumes" Oct 30 20:20:16.005798 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:20:16.005766 4959 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7053-44d415\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" pod="opendatahub/test-curl-7053-44d415" Oct 30 20:20:16.066390 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:20:16.066357 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:20:17.015352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:17.015306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:18.770968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:18.770935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:18.898369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:18.898330 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:19.007055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:19.007020 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.378976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.378943 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:20:20.379741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.379723 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" containerName="test-curl-7053-44d415" Oct 30 20:20:20.379815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.379741 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" containerName="test-curl-7053-44d415" Oct 30 20:20:20.379854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.379844 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf9fdac-5fdf-47a7-aff5-476258034291" containerName="test-curl-7053-44d415" Oct 30 20:20:20.384716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.384697 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.411190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.411153 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6175d5-serving-cert\"" Oct 30 20:20:20.414682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.414657 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:20:20.468010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.467978 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.501187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.501160 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.501349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.501287 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.501396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.501368 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvsw\" (UniqueName: \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.501456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.501442 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.578155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.578119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.602984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.602942 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.603350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.603123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.603350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.603199 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvsw\" (UniqueName: \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.603350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.603276 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.603813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.603791 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.606172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.606146 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.607006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.606980 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.624027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.624001 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvsw\" (UniqueName: \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\") pod \"maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:20.697094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.697057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.699144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:20.699115 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:21.007147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:21.007068 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:20:21.010812 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:20:21.010786 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e114e5_725a_44fe_b8ef_4edabae56e2d.slice/crio-c5a6efdaf9db134873f4be20102f4b1ac14bea5db84ca76c32107f928681ee1b WatchSource:0}: Error finding container c5a6efdaf9db134873f4be20102f4b1ac14bea5db84ca76c32107f928681ee1b: Status 404 returned error can't find the container with id c5a6efdaf9db134873f4be20102f4b1ac14bea5db84ca76c32107f928681ee1b Oct 30 20:20:21.957614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:21.957574 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" event={"ID":"71e114e5-725a-44fe-b8ef-4edabae56e2d","Type":"ContainerStarted","Data":"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9"} Oct 30 20:20:21.958077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:21.957629 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" event={"ID":"71e114e5-725a-44fe-b8ef-4edabae56e2d","Type":"ContainerStarted","Data":"c5a6efdaf9db134873f4be20102f4b1ac14bea5db84ca76c32107f928681ee1b"} Oct 30 20:20:21.958077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:21.957805 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:22.000161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:22.000108 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" podStartSLOduration=1.59461698 podStartE2EDuration="2.000091932s" totalImagesPullingTime="405.474952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:21.997742627 +0000 UTC m=+2746.126976549" watchObservedRunningTime="2026-10-30 20:20:22.000091932 +0000 UTC m=+2746.129325853" Oct 30 20:20:22.447300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:22.447255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:24.246398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:24.246337 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:25.964887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:25.964847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:26.244917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.244831 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:20:26.250059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.250034 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.276444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.276419 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"default-dockercfg-x46lw\"" Oct 30 20:20:26.276444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.276430 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"openshift-service-ca.crt\"" Oct 30 20:20:26.276665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.276429 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"kube-root-ca.crt\"" Oct 30 20:20:26.276665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.276475 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5c875d\"/\"auth-test-model-5c875d-kserve-self-signed-certs\"" Oct 30 20:20:26.280556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.280532 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:20:26.293884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.293858 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.294050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.293900 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.294050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.293928 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.294177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.294087 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.294177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.294136 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8d8x\" (UniqueName: \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.294310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.294182 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.386421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.386384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:26.395091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395063 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395115 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395231 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8d8x\" (UniqueName: \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395287 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395462 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395444 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395597 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395577 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395629 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395882 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.395996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.395977 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.397782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.397755 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.397956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.397938 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.415002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.414980 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8d8x\" (UniqueName: \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\") pod \"auth-test-model-5c875d-kserve-5f998898c7-9vb8q\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.576605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.576519 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:26.889707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.889685 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:20:26.893020 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:20:26.892979 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda46fd4b2_f220_4ef8_8ec8_7f0359247c4e.slice/crio-4f21da8e5673a3868ecee8119ff74cf1074ed40d7497f253d1d5bacabe5f34b8 WatchSource:0}: Error finding container 4f21da8e5673a3868ecee8119ff74cf1074ed40d7497f253d1d5bacabe5f34b8: Status 404 returned error can't find the container with id 4f21da8e5673a3868ecee8119ff74cf1074ed40d7497f253d1d5bacabe5f34b8 Oct 30 20:20:26.994537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.994254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" event={"ID":"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e","Type":"ContainerStarted","Data":"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f"} Oct 30 20:20:26.994537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:26.994308 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" event={"ID":"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e","Type":"ContainerStarted","Data":"4f21da8e5673a3868ecee8119ff74cf1074ed40d7497f253d1d5bacabe5f34b8"} Oct 30 20:20:27.036182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:27.036119 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" podStartSLOduration=1.036103894 podStartE2EDuration="1.036103894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:27.032602905 +0000 UTC m=+2751.161836825" watchObservedRunningTime="2026-10-30 20:20:27.036103894 +0000 UTC m=+2751.165337817" Oct 30 20:20:27.576921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:27.576891 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:27.713041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:27.713003 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:27.971094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:27.971066 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:20:29.490378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:29.490334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:31.287071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:31.287028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.604001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:32.603966 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:33.006181 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:33.006150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:34.799633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:34.799596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:36.577727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:36.577693 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:36.594933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:36.594905 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:36.622586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:36.622556 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:36.739272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:36.739239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:36.855732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:36.855637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:37.078641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:37.078616 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:20:38.480266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:38.480238 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:40.300548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:40.300489 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:42.041746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:42.041703 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:43.697586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:43.697538 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:45.455011 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:45.454974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:46.067368 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:20:46.067321 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:20:46.187688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:46.187649 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.548785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:47.548750 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:49.429886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:49.429858 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:51.289871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:51.289832 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.052756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:53.052717 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.219631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:55.219597 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:56.979800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:56.979761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:58.622358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:20:58.622324 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:00.414670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:00.414634 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:02.206070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:02.206034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:03.983849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:03.983814 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:05.776196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:05.776155 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:06.463213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:06.463176 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.062668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.062632 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.523013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.522977 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.610658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.610624 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:21:07.640278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.640245 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.640452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.640315 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:21:07.744826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.744764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.854969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:07.854868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:09.507710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:09.507669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.327471 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:11.327441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.443780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:11.443738 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:12.611769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:12.611707 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" podUID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" containerName="payload-processing" containerID="cri-o://79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46" gracePeriod=25 Oct 30 20:21:12.641630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:12.641560 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" podUID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" containerName="payload-pre-processing" containerID="cri-o://6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9" gracePeriod=25 Oct 30 20:21:12.926691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:12.926661 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:21:12.988441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:12.988415 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:21:13.036259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.036228 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\" (UniqueName: \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\") pod \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " Oct 30 20:21:13.036448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.036295 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\") pod \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\" (UID: \"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758\") " Oct 30 20:21:13.036729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.036704 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume" pod "fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" (UID: "fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:13.036998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.036980 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:13.037245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.037199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.038819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.038800 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5" pod "fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" (UID: "fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758"). InnerVolumeSpecName "kube-api-access-4kwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:13.137623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.137539 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\" (UniqueName: \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\") pod \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " Oct 30 20:21:13.137787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.137730 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\") pod \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\" (UID: \"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645\") " Oct 30 20:21:13.138126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.138096 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume" pod "93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" (UID: "93d80d4c-8615-4f5d-ac4e-5db9c3f4c645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:13.138354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.138333 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:13.138425 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.138358 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwd5\" (UniqueName: \"kubernetes.io/projected/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758-kube-api-access-4kwd5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:13.139948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.139927 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk" pod "93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" (UID: "93d80d4c-8615-4f5d-ac4e-5db9c3f4c645"). InnerVolumeSpecName "kube-api-access-mdqsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:13.189118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.189086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.239606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.239568 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqsk\" (UniqueName: \"kubernetes.io/projected/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645-kube-api-access-mdqsk\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:13.324665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.324632 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:21:13.325396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325375 4959 cpu_manager.go:414] "containerMap: removing container" podUID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" containerName="payload-pre-processing" Oct 30 20:21:13.325396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325394 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" containerName="payload-pre-processing" Oct 30 20:21:13.325625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325411 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" containerName="payload-processing" Oct 30 20:21:13.325625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325418 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" containerName="payload-processing" Oct 30 20:21:13.325625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325623 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" containerName="payload-pre-processing" Oct 30 20:21:13.325782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.325637 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" containerName="payload-processing" Oct 30 20:21:13.330473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.330219 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.352153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.352116 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fbda3-dockercfg-l2trt\"" Oct 30 20:21:13.363998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.363920 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fbda3\"" Oct 30 20:21:13.388059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.387933 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:21:13.394700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.394131 4959 generic.go:356] "Generic (PLEG): container finished" podID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" containerID="79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46" exitCode=0 Oct 30 20:21:13.394700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.394313 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" event={"ID":"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758","Type":"ContainerDied","Data":"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46"} Oct 30 20:21:13.394700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.394538 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" Oct 30 20:21:13.418522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.414074 4959 generic.go:356] "Generic (PLEG): container finished" podID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" containerID="6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9" exitCode=0 Oct 30 20:21:13.418522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.414347 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" Oct 30 20:21:13.444054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.442700 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfgb\" (UniqueName: \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.444054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.442755 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.446610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.445444 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk" event={"ID":"fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758","Type":"ContainerDied","Data":"f7b838a8dea5e2f058d9183790ff4e9659cca8004fd20df4292f8affe5a96eab"} Oct 30 20:21:13.446610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.445522 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" event={"ID":"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645","Type":"ContainerDied","Data":"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9"} Oct 30 20:21:13.446610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.445555 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l" event={"ID":"93d80d4c-8615-4f5d-ac4e-5db9c3f4c645","Type":"ContainerDied","Data":"a57bfc4c9464a67a3f4d1b782f7ae1029af08b9eea8e646d2ede8e1662371141"} Oct 30 20:21:13.446610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.445587 4959 scope.go:122] "RemoveContainer" containerID="79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46" Oct 30 20:21:13.449442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.449398 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:21:13.460863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.454715 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.469975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.469223 4959 scope.go:122] "RemoveContainer" containerID="79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46" Oct 30 20:21:13.470226 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:13.470204 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46\": container with ID starting with 79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46 not found: ID does not exist" containerID="79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46" Oct 30 20:21:13.470599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.470250 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46"} err="failed to get container status \"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46\": rpc error: code = NotFound desc = could not find container \"79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46\": container with ID starting with 79ab38d991e9724940d7a82428559bf8d2033db9ab2c5301322b0efac8d81b46 not found: ID does not exist" Oct 30 20:21:13.470599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.470271 4959 scope.go:122] "RemoveContainer" containerID="6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9" Oct 30 20:21:13.470768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.470663 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:21:13.484291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.484263 4959 scope.go:122] "RemoveContainer" containerID="6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9" Oct 30 20:21:13.484659 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:13.484636 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9\": container with ID starting with 6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9 not found: ID does not exist" containerID="6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9" Oct 30 20:21:13.484741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.484670 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9"} err="failed to get container status \"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9\": rpc error: code = NotFound desc = could not find container \"6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9\": container with ID starting with 6ef065b2950259cd3d48c42aa021f445dabd1c7cd0ff895f4c1f9a89c6d99bc9 not found: ID does not exist" Oct 30 20:21:13.533077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.533042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.534070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.534047 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:21:13.537447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.537424 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6175d5-5c55d548c6-cp8hk"] Oct 30 20:21:13.544776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.544753 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfgb\" (UniqueName: \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.544892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.544795 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.544961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.544922 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvtr\" (UniqueName: \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.545125 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.545101 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.545615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.545593 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.564186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.564160 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfgb\" (UniqueName: \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\") pod \"payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.577164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.577137 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:21:13.582388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.582352 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6175d5-575bc85587-np55l"] Oct 30 20:21:13.646310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.646236 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvtr\" (UniqueName: \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.646736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.646346 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.647126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.647105 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.664944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.664920 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvtr\" (UniqueName: \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\") pod \"payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.694418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.694393 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:13.779349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.779315 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:13.998798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.998719 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d80d4c-8615-4f5d-ac4e-5db9c3f4c645" path="/var/lib/kubelet/pods/93d80d4c-8615-4f5d-ac4e-5db9c3f4c645/volumes" Oct 30 20:21:13.999190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:13.999175 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758" path="/var/lib/kubelet/pods/fc3db2d5-0f91-4a7f-a0ca-1c0dfa77e758/volumes" Oct 30 20:21:14.010829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.010787 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:21:14.013753 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:21:14.013727 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a030b88_f0c1_4017_8f96_1e35c26414a3.slice/crio-edd02cea998dd1d9dec00e58bdc9301323b36a8695f0fb7e510dba3e6c45f488 WatchSource:0}: Error finding container edd02cea998dd1d9dec00e58bdc9301323b36a8695f0fb7e510dba3e6c45f488: Status 404 returned error can't find the container with id edd02cea998dd1d9dec00e58bdc9301323b36a8695f0fb7e510dba3e6c45f488 Oct 30 20:21:14.110196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.110167 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:21:14.113565 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:21:14.113537 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06296c9c_15fd_4326_981f_5c719dd5ea5d.slice/crio-3b0850f50db6d2f50a0ee0f676c29ef35afea4f608286f6692f9e4b47f26a1e6 WatchSource:0}: Error finding container 3b0850f50db6d2f50a0ee0f676c29ef35afea4f608286f6692f9e4b47f26a1e6: Status 404 returned error can't find the container with id 3b0850f50db6d2f50a0ee0f676c29ef35afea4f608286f6692f9e4b47f26a1e6 Oct 30 20:21:14.425087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.425049 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" event={"ID":"06296c9c-15fd-4326-981f-5c719dd5ea5d","Type":"ContainerStarted","Data":"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1"} Oct 30 20:21:14.425296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.425110 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" event={"ID":"06296c9c-15fd-4326-981f-5c719dd5ea5d","Type":"ContainerStarted","Data":"3b0850f50db6d2f50a0ee0f676c29ef35afea4f608286f6692f9e4b47f26a1e6"} Oct 30 20:21:14.425296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.425139 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:14.428634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.428608 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" event={"ID":"7a030b88-f0c1-4017-8f96-1e35c26414a3","Type":"ContainerStarted","Data":"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46"} Oct 30 20:21:14.428772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.428656 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" event={"ID":"7a030b88-f0c1-4017-8f96-1e35c26414a3","Type":"ContainerStarted","Data":"edd02cea998dd1d9dec00e58bdc9301323b36a8695f0fb7e510dba3e6c45f488"} Oct 30 20:21:14.428772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.428707 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:14.466369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.466310 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" podStartSLOduration=1.4662916 podStartE2EDuration="1.4662916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:14.463009357 +0000 UTC m=+2798.592243277" watchObservedRunningTime="2026-10-30 20:21:14.4662916 +0000 UTC m=+2798.595525522" Oct 30 20:21:14.504750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:14.504704 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" podStartSLOduration=1.504690917 podStartE2EDuration="1.504690917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:14.503949557 +0000 UTC m=+2798.633183478" watchObservedRunningTime="2026-10-30 20:21:14.504690917 +0000 UTC m=+2798.633924837" Oct 30 20:21:15.413891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:15.413856 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:16.065643 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:16.065607 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:21:17.329328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:17.329284 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:19.616101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:19.616063 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:21.494543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:21.494481 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:21.879647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:21.879558 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:22.146443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:22.146392 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:21:22.315443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:22.315384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:22.432932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:22.432840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:22.538110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:22.538077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:23.605802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:23.605744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.403121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:25.403078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.445890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:25.445859 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:21:25.447321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:25.447292 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:21:27.125146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.125113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:27.146738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.146699 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" podUID="c20e90aa-2296-4980-bd51-eb62f71444c7" containerName="payload-processing" containerID="cri-o://115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2" gracePeriod=25 Oct 30 20:21:27.472561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.472535 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:21:27.570132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.570091 4959 generic.go:356] "Generic (PLEG): container finished" podID="c20e90aa-2296-4980-bd51-eb62f71444c7" containerID="115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2" exitCode=0 Oct 30 20:21:27.570132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.570133 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" Oct 30 20:21:27.570404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.570169 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" event={"ID":"c20e90aa-2296-4980-bd51-eb62f71444c7","Type":"ContainerDied","Data":"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2"} Oct 30 20:21:27.570404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.570224 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9" event={"ID":"c20e90aa-2296-4980-bd51-eb62f71444c7","Type":"ContainerDied","Data":"f20a8d33f18c58d7c1409112f7fde1b99a1765701f2c44a64390b9d55ba66750"} Oct 30 20:21:27.570404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.570254 4959 scope.go:122] "RemoveContainer" containerID="115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2" Oct 30 20:21:27.582191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.582170 4959 scope.go:122] "RemoveContainer" containerID="115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2" Oct 30 20:21:27.582488 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:27.582469 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2\": container with ID starting with 115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2 not found: ID does not exist" containerID="115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2" Oct 30 20:21:27.582578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.582514 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2"} err="failed to get container status \"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2\": rpc error: code = NotFound desc = could not find container \"115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2\": container with ID starting with 115eae13e1e2d5a6cfcb37afc06550dd980e3cdef0c13136b1d9efc4859ccbb2 not found: ID does not exist" Oct 30 20:21:27.586373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.586348 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\") pod \"c20e90aa-2296-4980-bd51-eb62f71444c7\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " Oct 30 20:21:27.586456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.586439 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\" (UniqueName: \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\") pod \"c20e90aa-2296-4980-bd51-eb62f71444c7\" (UID: \"c20e90aa-2296-4980-bd51-eb62f71444c7\") " Oct 30 20:21:27.586711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.586686 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume" pod "c20e90aa-2296-4980-bd51-eb62f71444c7" (UID: "c20e90aa-2296-4980-bd51-eb62f71444c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:27.586941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.586907 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c20e90aa-2296-4980-bd51-eb62f71444c7-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:27.588708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.588689 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm" pod "c20e90aa-2296-4980-bd51-eb62f71444c7" (UID: "c20e90aa-2296-4980-bd51-eb62f71444c7"). InnerVolumeSpecName "kube-api-access-msjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:27.688355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.688288 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjhm\" (UniqueName: \"kubernetes.io/projected/c20e90aa-2296-4980-bd51-eb62f71444c7-kube-api-access-msjhm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:21:27.923305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.916179 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:21:27.923305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:27.919615 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30e92c-7d5d4ccddb-kcdq9"] Oct 30 20:21:28.001300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.001213 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20e90aa-2296-4980-bd51-eb62f71444c7" path="/var/lib/kubelet/pods/c20e90aa-2296-4980-bd51-eb62f71444c7/volumes" Oct 30 20:21:28.578654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.578625 4959 generic.go:356] "Generic (PLEG): container finished" podID="618a7097-47ef-4d5b-8474-4353dface889" containerID="42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0" exitCode=0 Oct 30 20:21:28.579140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.578703 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerDied","Data":"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0"} Oct 30 20:21:28.579228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.579209 4959 scope.go:122] "RemoveContainer" containerID="42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0" Oct 30 20:21:28.581219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.581190 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:21:28.581289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.581265 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:21:28.581331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.581298 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:21:28.886308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:28.886276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:29.589860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:29.589823 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerStarted","Data":"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020"} Oct 30 20:21:29.619029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:29.618997 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:21:29.619198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:29.619180 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:29.619248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:29.619233 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:30.520802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:30.520766 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:30.619711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:30.619668 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:30.620282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:30.619739 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:31.620026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:31.619988 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:31.620457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:31.620047 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:32.221804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:32.221767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:32.619279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:32.619204 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:32.619279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:32.619252 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:32.840062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:32.840028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:33.270521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:33.259991 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:33.619217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:33.619134 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:33.619217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:33.619189 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:33.992868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:33.992829 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:34.619479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:34.619441 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 20:21:34.619717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:34.619545 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 20:21:35.624582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:35.624541 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:21:35.625054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:35.625032 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:21:35.625809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:35.625785 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:21:35.739849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:35.739804 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:38.016696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:38.016665 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:39.690303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:39.690269 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:41.322090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:41.322057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:43.115113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:43.115085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:44.985117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:44.985083 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.185529 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:46.182974 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:21:46.360686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:46.351009 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.521807 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:46.521690 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:47.453666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:47.453619 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:49.119019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:49.118985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:50.909361 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:50.909328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:52.709921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:52.709883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.267933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.267901 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:21:54.268595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.268580 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c20e90aa-2296-4980-bd51-eb62f71444c7" containerName="payload-processing" Oct 30 20:21:54.268654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.268595 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20e90aa-2296-4980-bd51-eb62f71444c7" containerName="payload-processing" Oct 30 20:21:54.268750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.268741 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20e90aa-2296-4980-bd51-eb62f71444c7" containerName="payload-processing" Oct 30 20:21:54.273973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.273954 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.293941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.293913 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5fbda3-serving-cert\"" Oct 30 20:21:54.297951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.297930 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:21:54.347017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.346988 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.347170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.347061 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.347229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.347204 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.347268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.347244 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgd7\" (UniqueName: \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.351056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.351032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.454096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.450416 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.454486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.454438 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.454715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.454692 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.454847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.454776 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgd7\" (UniqueName: \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.455031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.455008 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.455611 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:54.455206 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-serving-cert: secret "maas-api-e2e-ipp-a-5fbda3-serving-cert" not found Oct 30 20:21:54.455847 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:21:54.455833 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls podName:35c9f14a-fae3-4dc8-bfba-edb439bd763a nodeName:}" failed. No retries permitted until 2026-10-30 20:21:54.955810693 +0000 UTC m=+2839.085044606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls") pod "maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" (UID: "35c9f14a-fae3-4dc8-bfba-edb439bd763a") : secret "maas-api-e2e-ipp-a-5fbda3-serving-cert" not found Oct 30 20:21:54.459264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.459234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.460071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.460040 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.476565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.476534 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgd7\" (UniqueName: \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.701621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.701586 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.962162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.962077 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:54.964590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:54.964571 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:55.188678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:55.188644 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:55.496565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:55.496463 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:21:55.500822 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:21:55.500794 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c9f14a_fae3_4dc8_bfba_edb439bd763a.slice/crio-a3a9cb2e33fcda6289aa21180609982534a90bc26fff04279d9bf1755d76b813 WatchSource:0}: Error finding container a3a9cb2e33fcda6289aa21180609982534a90bc26fff04279d9bf1755d76b813: Status 404 returned error can't find the container with id a3a9cb2e33fcda6289aa21180609982534a90bc26fff04279d9bf1755d76b813 Oct 30 20:21:55.502368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:55.502347 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:21:55.794826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:55.794741 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" event={"ID":"35c9f14a-fae3-4dc8-bfba-edb439bd763a","Type":"ContainerStarted","Data":"a3a9cb2e33fcda6289aa21180609982534a90bc26fff04279d9bf1755d76b813"} Oct 30 20:21:56.119088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:56.118992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:56.803742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:56.803698 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" event={"ID":"35c9f14a-fae3-4dc8-bfba-edb439bd763a","Type":"ContainerStarted","Data":"874fd9db07e778a9cde3eafbf6868ad568cd022161d315cbf78ba16db28acbcb"} Oct 30 20:21:56.804215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:56.803781 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:21:56.844521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:56.844438 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" podStartSLOduration=2.508740323 podStartE2EDuration="2.84442439s" totalImagesPullingTime="335.684067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:56.843245213 +0000 UTC m=+2840.972479148" watchObservedRunningTime="2026-10-30 20:21:56.84442439 +0000 UTC m=+2840.973658311" Oct 30 20:21:57.857674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:57.857622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:59.589341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:21:59.589304 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.192654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:00.192620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.844111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:00.844060 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:01.212846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:01.212810 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:01.602123 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:01.602044 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:02.817347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:02.817318 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:22:03.490784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:03.490745 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:04.865776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:04.865741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:05.008301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:05.008259 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:05.124536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:05.124421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.225627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.225593 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.599956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.599849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.710719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.710679 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:22:06.715100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.715079 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.716969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.716933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:06.722227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.722200 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkg6c\" (UniqueName: \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.722478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.722460 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.734214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.734193 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:22:06.740819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.740782 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7eec8\"" Oct 30 20:22:06.771371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.771335 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:22:06.775852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.775828 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.791595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.791571 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7eec8-dockercfg-gscbz\"" Oct 30 20:22:06.794015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.793990 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:22:06.824480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.824441 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.825624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.824604 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl65\" (UniqueName: \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.825624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.824642 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkg6c\" (UniqueName: \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.825624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.824801 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.825624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.825260 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.851029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.850955 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkg6c\" (UniqueName: \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\") pod \"payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:06.926520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.926274 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl65\" (UniqueName: \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.926792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.926724 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.929564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.927626 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:06.945718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:06.945687 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl65\" (UniqueName: \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\") pod \"payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:07.030159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.030120 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:07.089760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.089727 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:07.337133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.337107 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:22:07.341417 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:22:07.341386 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cc77aa_237e_4b61_ac5d_9a92af92630f.slice/crio-eafb617fd4b879d2a09b8a57955b9c0f7b49fb21c2c2f537d50efd1729ff7696 WatchSource:0}: Error finding container eafb617fd4b879d2a09b8a57955b9c0f7b49fb21c2c2f537d50efd1729ff7696: Status 404 returned error can't find the container with id eafb617fd4b879d2a09b8a57955b9c0f7b49fb21c2c2f537d50efd1729ff7696 Oct 30 20:22:07.419119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.419088 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:22:07.422777 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:22:07.422749 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dd2bc6_0a7f_4aad_a0bc_dc54895b24cd.slice/crio-b7652c41677027b46381929b4a92e983219e7ebbd0cf8b95ab3807885c5b9ad8 WatchSource:0}: Error finding container b7652c41677027b46381929b4a92e983219e7ebbd0cf8b95ab3807885c5b9ad8: Status 404 returned error can't find the container with id b7652c41677027b46381929b4a92e983219e7ebbd0cf8b95ab3807885c5b9ad8 Oct 30 20:22:07.901432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.901399 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" event={"ID":"27cc77aa-237e-4b61-ac5d-9a92af92630f","Type":"ContainerStarted","Data":"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9"} Oct 30 20:22:07.901680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.901451 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" event={"ID":"27cc77aa-237e-4b61-ac5d-9a92af92630f","Type":"ContainerStarted","Data":"eafb617fd4b879d2a09b8a57955b9c0f7b49fb21c2c2f537d50efd1729ff7696"} Oct 30 20:22:07.901680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.901643 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:07.903069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.903047 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" event={"ID":"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd","Type":"ContainerStarted","Data":"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66"} Oct 30 20:22:07.903162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.903082 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" event={"ID":"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd","Type":"ContainerStarted","Data":"b7652c41677027b46381929b4a92e983219e7ebbd0cf8b95ab3807885c5b9ad8"} Oct 30 20:22:07.903162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.903138 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:07.944922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.944852 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" podStartSLOduration=1.944834938 podStartE2EDuration="1.944834938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:07.940700107 +0000 UTC m=+2852.069934027" watchObservedRunningTime="2026-10-30 20:22:07.944834938 +0000 UTC m=+2852.074068858" Oct 30 20:22:07.980885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.980806 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" podStartSLOduration=1.980792949 podStartE2EDuration="1.980792949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:07.977734229 +0000 UTC m=+2852.106968147" watchObservedRunningTime="2026-10-30 20:22:07.980792949 +0000 UTC m=+2852.110026864" Oct 30 20:22:07.991637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:07.991609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:09.790854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:09.790813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:11.512644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:11.512599 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.224672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:13.224632 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:14.870754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:14.870712 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:16.067210 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:16.067175 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:22:16.583141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:16.583105 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.309795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:18.309761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.913105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:18.913076 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:22:18.913538 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:18.913486 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:22:19.992643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:19.992609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:22.195429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:22.195396 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:22.751020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:22.750985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:24.087417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:24.087363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:25.833829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:25.833796 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.480038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:27.480002 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:29.322919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:29.322878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:31.595840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:31.595803 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:32.843747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:32.843701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:33.226470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:33.226425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:33.409843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:33.409804 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.414177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:35.414128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:37.355910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:37.355873 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:39.386204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:39.386168 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.301633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:41.301594 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.879450 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:41.879417 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.988791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:41.988755 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.106901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.106861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.391344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.391314 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:22:43.424240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.424208 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:22:43.533253 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.533211 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.643184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.643092 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:43.744419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:43.744387 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:44.924669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:44.924640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:46.070615 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:46.070577 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:22:46.828328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:46.828295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.391877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.391835 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" podUID="7a030b88-f0c1-4017-8f96-1e35c26414a3" containerName="payload-processing" containerID="cri-o://cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46" gracePeriod=25 Oct 30 20:22:48.424697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.424663 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" podUID="06296c9c-15fd-4326-981f-5c719dd5ea5d" containerName="payload-pre-processing" containerID="cri-o://caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1" gracePeriod=25 Oct 30 20:22:48.675364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.675276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.741675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.741639 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:22:48.771353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.771328 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:22:48.910483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.910394 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\" (UniqueName: \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\") pod \"7a030b88-f0c1-4017-8f96-1e35c26414a3\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " Oct 30 20:22:48.910483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.910439 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\" (UniqueName: \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\") pod \"06296c9c-15fd-4326-981f-5c719dd5ea5d\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " Oct 30 20:22:48.910769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.910564 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\") pod \"06296c9c-15fd-4326-981f-5c719dd5ea5d\" (UID: \"06296c9c-15fd-4326-981f-5c719dd5ea5d\") " Oct 30 20:22:48.910769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.910643 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\") pod \"7a030b88-f0c1-4017-8f96-1e35c26414a3\" (UID: \"7a030b88-f0c1-4017-8f96-1e35c26414a3\") " Oct 30 20:22:48.911111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.911033 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume" pod "06296c9c-15fd-4326-981f-5c719dd5ea5d" (UID: "06296c9c-15fd-4326-981f-5c719dd5ea5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:48.911401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.911171 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume" pod "7a030b88-f0c1-4017-8f96-1e35c26414a3" (UID: "7a030b88-f0c1-4017-8f96-1e35c26414a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:48.911401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.911347 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06296c9c-15fd-4326-981f-5c719dd5ea5d-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:22:48.912964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.912940 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb" pod "7a030b88-f0c1-4017-8f96-1e35c26414a3" (UID: "7a030b88-f0c1-4017-8f96-1e35c26414a3"). InnerVolumeSpecName "kube-api-access-vdfgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:48.913170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:48.913150 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr" pod "06296c9c-15fd-4326-981f-5c719dd5ea5d" (UID: "06296c9c-15fd-4326-981f-5c719dd5ea5d"). InnerVolumeSpecName "kube-api-access-qtvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:49.012315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.012280 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a030b88-f0c1-4017-8f96-1e35c26414a3-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:22:49.012315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.012312 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfgb\" (UniqueName: \"kubernetes.io/projected/7a030b88-f0c1-4017-8f96-1e35c26414a3-kube-api-access-vdfgb\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:22:49.012315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.012323 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtvtr\" (UniqueName: \"kubernetes.io/projected/06296c9c-15fd-4326-981f-5c719dd5ea5d-kube-api-access-qtvtr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:22:49.205536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.205444 4959 generic.go:356] "Generic (PLEG): container finished" podID="06296c9c-15fd-4326-981f-5c719dd5ea5d" containerID="caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1" exitCode=0 Oct 30 20:22:49.205536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.205489 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" Oct 30 20:22:49.205745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.205535 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" event={"ID":"06296c9c-15fd-4326-981f-5c719dd5ea5d","Type":"ContainerDied","Data":"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1"} Oct 30 20:22:49.205745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.205589 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667" event={"ID":"06296c9c-15fd-4326-981f-5c719dd5ea5d","Type":"ContainerDied","Data":"3b0850f50db6d2f50a0ee0f676c29ef35afea4f608286f6692f9e4b47f26a1e6"} Oct 30 20:22:49.205745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.205613 4959 scope.go:122] "RemoveContainer" containerID="caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1" Oct 30 20:22:49.207177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.207157 4959 generic.go:356] "Generic (PLEG): container finished" podID="7a030b88-f0c1-4017-8f96-1e35c26414a3" containerID="cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46" exitCode=0 Oct 30 20:22:49.207308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.207196 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" Oct 30 20:22:49.207308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.207283 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" event={"ID":"7a030b88-f0c1-4017-8f96-1e35c26414a3","Type":"ContainerDied","Data":"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46"} Oct 30 20:22:49.207415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.207321 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m" event={"ID":"7a030b88-f0c1-4017-8f96-1e35c26414a3","Type":"ContainerDied","Data":"edd02cea998dd1d9dec00e58bdc9301323b36a8695f0fb7e510dba3e6c45f488"} Oct 30 20:22:49.217079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.217061 4959 scope.go:122] "RemoveContainer" containerID="caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1" Oct 30 20:22:49.217399 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:49.217372 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1\": container with ID starting with caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1 not found: ID does not exist" containerID="caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1" Oct 30 20:22:49.217478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.217413 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1"} err="failed to get container status \"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1\": rpc error: code = NotFound desc = could not find container \"caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1\": container with ID starting with caa7841bf8bf4a33fa0c25e72bed3d427c83052d4116b3e148ac4bef58841cc1 not found: ID does not exist" Oct 30 20:22:49.217478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.217437 4959 scope.go:122] "RemoveContainer" containerID="cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46" Oct 30 20:22:49.226978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.226960 4959 scope.go:122] "RemoveContainer" containerID="cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46" Oct 30 20:22:49.227250 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:49.227230 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46\": container with ID starting with cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46 not found: ID does not exist" containerID="cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46" Oct 30 20:22:49.227295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.227260 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46"} err="failed to get container status \"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46\": rpc error: code = NotFound desc = could not find container \"cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46\": container with ID starting with cc9f105c36f8af065304dffe7205c7b62a8f514cbf5948219ccae8808859cb46 not found: ID does not exist" Oct 30 20:22:49.253882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.253853 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:22:49.264561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.264539 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbda3-6dc64cf699-lr667"] Oct 30 20:22:49.288382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.288349 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:22:49.293632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.293609 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbda3-55769bdd74-8pz8m"] Oct 30 20:22:49.999724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:49.999681 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06296c9c-15fd-4326-981f-5c719dd5ea5d" path="/var/lib/kubelet/pods/06296c9c-15fd-4326-981f-5c719dd5ea5d/volumes" Oct 30 20:22:50.000283 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.000267 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a030b88-f0c1-4017-8f96-1e35c26414a3" path="/var/lib/kubelet/pods/7a030b88-f0c1-4017-8f96-1e35c26414a3/volumes" Oct 30 20:22:50.868058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868020 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:22:50.868747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868732 4959 cpu_manager.go:414] "containerMap: removing container" podUID="06296c9c-15fd-4326-981f-5c719dd5ea5d" containerName="payload-pre-processing" Oct 30 20:22:50.868806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868749 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06296c9c-15fd-4326-981f-5c719dd5ea5d" containerName="payload-pre-processing" Oct 30 20:22:50.868806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868802 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7a030b88-f0c1-4017-8f96-1e35c26414a3" containerName="payload-processing" Oct 30 20:22:50.868887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868807 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a030b88-f0c1-4017-8f96-1e35c26414a3" containerName="payload-processing" Oct 30 20:22:50.868920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868904 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a030b88-f0c1-4017-8f96-1e35c26414a3" containerName="payload-processing" Oct 30 20:22:50.868920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.868914 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="06296c9c-15fd-4326-981f-5c719dd5ea5d" containerName="payload-pre-processing" Oct 30 20:22:50.874305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.874281 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:50.891655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.891624 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:22:50.898847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.898818 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b7eec8-serving-cert\"" Oct 30 20:22:50.958867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:50.958837 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:51.040072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.040043 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.040481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.040161 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.040481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.040323 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.040481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.040376 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cdg\" (UniqueName: \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.075806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.075777 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:51.142220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.142193 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.142433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.142276 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.142433 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:51.142355 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-serving-cert: secret "maas-api-e2e-ipp-b-b7eec8-serving-cert" not found Oct 30 20:22:51.142433 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:22:51.142427 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls podName:fee95dd6-e2a1-437d-a529-d1a50e4096ad nodeName:}" failed. No retries permitted until 2026-10-30 20:22:51.642406704 +0000 UTC m=+2895.771640620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls") pod "maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" (UID: "fee95dd6-e2a1-437d-a529-d1a50e4096ad") : secret "maas-api-e2e-ipp-b-b7eec8-serving-cert" not found Oct 30 20:22:51.142655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.142451 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.142655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.142558 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cdg\" (UniqueName: \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.143117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.143095 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.145255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.145231 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.161683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.161659 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cdg\" (UniqueName: \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.226694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.226653 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:51.650273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.650241 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.652936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.652907 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:51.788739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:51.788711 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:52.099880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:52.099777 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:22:52.108367 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:22:52.108330 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee95dd6_e2a1_437d_a529_d1a50e4096ad.slice/crio-38bfef699a70ed2a154ef85c82a7d9e917009def798436acf4f6396d49f748d7 WatchSource:0}: Error finding container 38bfef699a70ed2a154ef85c82a7d9e917009def798436acf4f6396d49f748d7: Status 404 returned error can't find the container with id 38bfef699a70ed2a154ef85c82a7d9e917009def798436acf4f6396d49f748d7 Oct 30 20:22:52.231457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:52.231422 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" event={"ID":"fee95dd6-e2a1-437d-a529-d1a50e4096ad","Type":"ContainerStarted","Data":"38bfef699a70ed2a154ef85c82a7d9e917009def798436acf4f6396d49f748d7"} Oct 30 20:22:52.986072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:52.986036 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:53.241357 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:53.241264 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" event={"ID":"fee95dd6-e2a1-437d-a529-d1a50e4096ad","Type":"ContainerStarted","Data":"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a"} Oct 30 20:22:53.241752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:53.241359 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:22:53.281608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:53.281554 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" podStartSLOduration=2.824613127 podStartE2EDuration="3.281539827s" totalImagesPullingTime="456.9267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:53.279436832 +0000 UTC m=+2897.408670763" watchObservedRunningTime="2026-10-30 20:22:53.281539827 +0000 UTC m=+2897.410773748" Oct 30 20:22:54.781191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:54.781144 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:56.509915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:56.509880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:58.222259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:58.222208 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:59.256837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:22:59.256803 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:23:00.007819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:00.007786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:01.680220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:01.680188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.396236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:03.396200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:05.609688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:05.609658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:07.341814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:07.341780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:09.628234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:09.628200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:11.427108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:11.427078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:13.234273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:13.234238 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:14.950030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:14.949993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:16.068627 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:23:16.068583 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:23:16.717485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:16.717444 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.801122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.801089 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:23:17.805866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.805843 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.819812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.819793 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-159b3a-openshift-default-dockercfg-8tw4f\"" Oct 30 20:23:17.835984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.835959 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:23:17.930330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930291 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930434 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930481 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft992\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930519 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930598 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930643 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930700 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930775 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:17.930889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:17.930869 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.031766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.031735 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.031924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.031817 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.031924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.031901 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.031946 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft992\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.031996 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032126 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032246 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032284 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032292 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032525 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.032708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032685 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.033011 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.032984 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.033184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.033165 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.034383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.034361 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.034941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.034924 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.050781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.050754 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.050984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.050968 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft992\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\") pod \"e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.121635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.121564 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:18.447959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.447925 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:23:18.451843 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:23:18.451813 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbafd718_e34e_40fb_922e_b648acfff217.slice/crio-47e03676b19f4f910f47f3562d1619d2f013e266acb9942f10f40abd19b41364 WatchSource:0}: Error finding container 47e03676b19f4f910f47f3562d1619d2f013e266acb9942f10f40abd19b41364: Status 404 returned error can't find the container with id 47e03676b19f4f910f47f3562d1619d2f013e266acb9942f10f40abd19b41364 Oct 30 20:23:18.454407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.454377 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:23:18.454474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.454442 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:23:18.454543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:18.454471 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:23:19.007427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:19.007395 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:19.475530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:19.475479 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" event={"ID":"dbafd718-e34e-40fb-922e-b648acfff217","Type":"ContainerStarted","Data":"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41"} Oct 30 20:23:19.475821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:19.475554 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" event={"ID":"dbafd718-e34e-40fb-922e-b648acfff217","Type":"ContainerStarted","Data":"47e03676b19f4f910f47f3562d1619d2f013e266acb9942f10f40abd19b41364"} Oct 30 20:23:19.527975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:19.527904 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podStartSLOduration=2.527884812 podStartE2EDuration="2.527884812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:19.525487628 +0000 UTC m=+2923.654721549" watchObservedRunningTime="2026-10-30 20:23:19.527884812 +0000 UTC m=+2923.657118738" Oct 30 20:23:20.122259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:20.122231 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:20.123845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:20.123821 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:20.123983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:20.123868 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:20.627395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:20.627360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:21.122671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:21.122639 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:21.123081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:21.122693 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:22.122188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:22.122160 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:22.122405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:22.122206 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:22.290237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:22.290206 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:23.122570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:23.122535 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:23.122787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:23.122588 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:24.040214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:24.040180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:24.122191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:24.122162 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:24.122368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:24.122215 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:25.122215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:25.122180 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 20:23:25.122744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:25.122232 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 20:23:25.973808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:25.973780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:26.127827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:26.127794 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:26.128319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:26.128287 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:26.129243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:26.129217 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:23:27.287149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:27.287109 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:27.454507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:27.454456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:28.635206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:28.635177 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:30.496463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:30.496424 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:32.127884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:32.127854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:32.949693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:32.949661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.341256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:33.341157 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.784085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:33.784046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:35.696618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:35.696587 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:37.995690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:37.995645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:40.040764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:40.040732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:41.459429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:41.459393 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:41.953136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:41.953095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:41.954547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:41.954527 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:23:41.983724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:41.983694 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:23:42.084152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:42.084117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:42.198269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:42.198213 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:42.319256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:42.318421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:44.012867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:44.012835 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:45.881751 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:45.881713 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:46.066551 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:23:46.066479 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:23:46.363755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:46.363727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:46.480722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:46.480688 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:46.956072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:46.956018 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" podUID="27cc77aa-237e-4b61-ac5d-9a92af92630f" containerName="payload-processing" containerID="cri-o://4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9" gracePeriod=25 Oct 30 20:23:46.984571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:46.984536 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" podUID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" containerName="payload-pre-processing" containerID="cri-o://17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66" gracePeriod=25 Oct 30 20:23:47.265391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.265363 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:23:47.332149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.332114 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:23:47.381340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.381309 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\") pod \"27cc77aa-237e-4b61-ac5d-9a92af92630f\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " Oct 30 20:23:47.381581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.381372 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\" (UniqueName: \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\") pod \"27cc77aa-237e-4b61-ac5d-9a92af92630f\" (UID: \"27cc77aa-237e-4b61-ac5d-9a92af92630f\") " Oct 30 20:23:47.381806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.381780 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume" pod "27cc77aa-237e-4b61-ac5d-9a92af92630f" (UID: "27cc77aa-237e-4b61-ac5d-9a92af92630f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:47.382142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.382122 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cc77aa-237e-4b61-ac5d-9a92af92630f-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:23:47.383721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.383700 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c" pod "27cc77aa-237e-4b61-ac5d-9a92af92630f" (UID: "27cc77aa-237e-4b61-ac5d-9a92af92630f"). InnerVolumeSpecName "kube-api-access-rkg6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:47.482861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.482767 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\" (UniqueName: \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\") pod \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " Oct 30 20:23:47.482861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.482857 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\") pod \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\" (UID: \"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd\") " Oct 30 20:23:47.483296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.483268 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume" pod "a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" (UID: "a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:47.483421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.483392 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:23:47.483421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.483411 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkg6c\" (UniqueName: \"kubernetes.io/projected/27cc77aa-237e-4b61-ac5d-9a92af92630f-kube-api-access-rkg6c\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:23:47.485216 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.485189 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65" pod "a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" (UID: "a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd"). InnerVolumeSpecName "kube-api-access-ljl65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:47.584504 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.584454 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljl65\" (UniqueName: \"kubernetes.io/projected/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd-kube-api-access-ljl65\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:23:47.685550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.685514 4959 generic.go:356] "Generic (PLEG): container finished" podID="27cc77aa-237e-4b61-ac5d-9a92af92630f" containerID="4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9" exitCode=0 Oct 30 20:23:47.685748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.685573 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" Oct 30 20:23:47.685748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.685615 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" event={"ID":"27cc77aa-237e-4b61-ac5d-9a92af92630f","Type":"ContainerDied","Data":"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9"} Oct 30 20:23:47.685748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.685676 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6" event={"ID":"27cc77aa-237e-4b61-ac5d-9a92af92630f","Type":"ContainerDied","Data":"eafb617fd4b879d2a09b8a57955b9c0f7b49fb21c2c2f537d50efd1729ff7696"} Oct 30 20:23:47.685748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.685705 4959 scope.go:122] "RemoveContainer" containerID="4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9" Oct 30 20:23:47.687692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.687649 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" Oct 30 20:23:47.687835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.687703 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" event={"ID":"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd","Type":"ContainerDied","Data":"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66"} Oct 30 20:23:47.687835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.687651 4959 generic.go:356] "Generic (PLEG): container finished" podID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" containerID="17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66" exitCode=0 Oct 30 20:23:47.687835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.687757 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5" event={"ID":"a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd","Type":"ContainerDied","Data":"b7652c41677027b46381929b4a92e983219e7ebbd0cf8b95ab3807885c5b9ad8"} Oct 30 20:23:47.706654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.706634 4959 scope.go:122] "RemoveContainer" containerID="4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9" Oct 30 20:23:47.706975 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:23:47.706945 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9\": container with ID starting with 4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9 not found: ID does not exist" containerID="4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9" Oct 30 20:23:47.707089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.706979 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9"} err="failed to get container status \"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9\": rpc error: code = NotFound desc = could not find container \"4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9\": container with ID starting with 4a8af4c857a7dd024bdde0753de3e0770f58feff0d824f8716e099464f0837f9 not found: ID does not exist" Oct 30 20:23:47.707089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.707002 4959 scope.go:122] "RemoveContainer" containerID="17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66" Oct 30 20:23:47.717739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.717720 4959 scope.go:122] "RemoveContainer" containerID="17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66" Oct 30 20:23:47.717994 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:23:47.717977 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66\": container with ID starting with 17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66 not found: ID does not exist" containerID="17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66" Oct 30 20:23:47.718076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.718001 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66"} err="failed to get container status \"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66\": rpc error: code = NotFound desc = could not find container \"17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66\": container with ID starting with 17691507dc5c377b495fbfc197ee42e7e19ce7ecd5904cce95faa7702357aa66 not found: ID does not exist" Oct 30 20:23:47.755956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.755872 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:23:47.759060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.759036 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b7eec8-6867657cc8-jsvv6"] Oct 30 20:23:47.791029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.790988 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:23:47.801793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.801765 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7eec8-846b6f78dc-gfrl5"] Oct 30 20:23:47.972487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:47.972447 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:48.002323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.002291 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27cc77aa-237e-4b61-ac5d-9a92af92630f" path="/var/lib/kubelet/pods/27cc77aa-237e-4b61-ac5d-9a92af92630f/volumes" Oct 30 20:23:48.002881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.002856 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" path="/var/lib/kubelet/pods/a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd/volumes" Oct 30 20:23:48.122418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.122319 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:48.230022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.229987 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:48.234804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.234769 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:23:48.235945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.235918 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" containerName="payload-pre-processing" Oct 30 20:23:48.236106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.236082 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" containerName="payload-pre-processing" Oct 30 20:23:48.236215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.236146 4959 cpu_manager.go:414] "containerMap: removing container" podUID="27cc77aa-237e-4b61-ac5d-9a92af92630f" containerName="payload-processing" Oct 30 20:23:48.236215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.236154 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27cc77aa-237e-4b61-ac5d-9a92af92630f" containerName="payload-processing" Oct 30 20:23:48.236355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.236344 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dd2bc6-0a7f-4aad-a0bc-dc54895b24cd" containerName="payload-pre-processing" Oct 30 20:23:48.236399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.236360 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="27cc77aa-237e-4b61-ac5d-9a92af92630f" containerName="payload-processing" Oct 30 20:23:48.241888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.241870 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.259404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.259194 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f35cc1\"" Oct 30 20:23:48.260871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.260847 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:23:48.288263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.288217 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:23:48.292836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.292809 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4tl\" (UniqueName: \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.292985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.292960 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.293895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.293876 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.315993 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.315961 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f35cc1-dockercfg-gcccb\"" Oct 30 20:23:48.321555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.321524 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:23:48.403574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.399857 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.403574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.399922 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.403574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.399982 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf2h\" (UniqueName: \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.403574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.400117 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4tl\" (UniqueName: \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.462242 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.462209 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.479873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.478075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4tl\" (UniqueName: \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\") pod \"payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.501208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.501174 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.501441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.501228 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf2h\" (UniqueName: \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.502312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.502282 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.523697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.523655 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf2h\" (UniqueName: \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\") pod \"payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.561667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.561624 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:48.626193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.626144 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:48.896468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.896439 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:23:48.897528 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:23:48.897463 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7291d7dd_9676_440a_9b38_99eb02326cd1.slice/crio-405d0f4571ca255372c5731260aee0b751ff5afc3d66a758f45d89fc74218cd0 WatchSource:0}: Error finding container 405d0f4571ca255372c5731260aee0b751ff5afc3d66a758f45d89fc74218cd0: Status 404 returned error can't find the container with id 405d0f4571ca255372c5731260aee0b751ff5afc3d66a758f45d89fc74218cd0 Oct 30 20:23:48.987058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:48.972816 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:23:49.720037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.719999 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" event={"ID":"fc92b9a6-e284-450a-becd-e88ba64484dc","Type":"ContainerStarted","Data":"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23"} Oct 30 20:23:49.720325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.720072 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" event={"ID":"fc92b9a6-e284-450a-becd-e88ba64484dc","Type":"ContainerStarted","Data":"f32298ced86996a6a7d599c140f938806b3ef679c47013ddf7e765df94f35fa7"} Oct 30 20:23:49.720325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.720119 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:23:49.721922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.721901 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" event={"ID":"7291d7dd-9676-440a-9b38-99eb02326cd1","Type":"ContainerStarted","Data":"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d"} Oct 30 20:23:49.722033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.721948 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:23:49.722033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.721969 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" event={"ID":"7291d7dd-9676-440a-9b38-99eb02326cd1","Type":"ContainerStarted","Data":"405d0f4571ca255372c5731260aee0b751ff5afc3d66a758f45d89fc74218cd0"} Oct 30 20:23:49.761349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.761286 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" podStartSLOduration=1.7612690720000002 podStartE2EDuration="1.761269072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:49.758587885 +0000 UTC m=+2953.887821818" watchObservedRunningTime="2026-10-30 20:23:49.761269072 +0000 UTC m=+2953.890502994" Oct 30 20:23:49.798351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.798297 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" podStartSLOduration=1.798281155 podStartE2EDuration="1.798281155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:49.797021924 +0000 UTC m=+2953.926255845" watchObservedRunningTime="2026-10-30 20:23:49.798281155 +0000 UTC m=+2953.927515079" Oct 30 20:23:49.835857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:49.835827 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:50.725083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:50.725045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:51.784576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:51.784546 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:53.574410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:53.574378 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:55.530866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:55.530830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:57.695790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:23:57.695755 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:00.030793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:00.030754 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:00.732101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:00.732060 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:24:00.732314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:00.732142 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:24:01.876815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:01.876766 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:03.388023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:03.387975 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:03.983577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:03.983543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:04.841727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:04.841694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:06.052691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:06.052658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:08.572965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:08.572921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:10.419865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:10.419811 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:12.316658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:12.316624 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:14.318668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:14.318631 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:15.507586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:15.507553 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 20:24:15.545188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:15.545156 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 20:24:15.652380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:15.652347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:15.788028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:15.787940 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:16.069113 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:16.069029 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:24:16.513631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:16.513601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:18.792967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:18.792933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:19.892369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:19.892338 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:20.508152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:20.508117 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" containerID="cri-o://ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5" gracePeriod=25 Oct 30 20:24:20.527391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:20.527358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:20.545448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:20.545409 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" containerID="cri-o://9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e" gracePeriod=25 Oct 30 20:24:20.777718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:20.777636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:21.047829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:21.047742 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:21.337837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:21.337756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:22.340537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:22.340477 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:22.713007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:22.712971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:23.332111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:23.332069 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:24.518303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:24.518263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:25.007324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:25.007285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:25.147307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:25.147267 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.616724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:26.616688 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.740912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:26.740878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.898897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:26.898863 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:26.977145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:26.977102 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:24:26.985072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:26.985040 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.014026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.013994 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c19563\"" Oct 30 20:24:27.016433 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.016405 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c19563-dockercfg-wtxz2\"" Oct 30 20:24:27.067718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.066637 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:24:27.150948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.150852 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwckp\" (UniqueName: \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.150948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.150926 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.253051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.253009 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.253343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.253325 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwckp\" (UniqueName: \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.253878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.253860 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.273461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.273428 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwckp\" (UniqueName: \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\") pod \"payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.360800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.360754 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:27.671739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:27.671697 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:24:27.674598 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:24:27.674571 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93386db5_1aeb_4d49_9efc_92113dc725ae.slice/crio-10b2e6b07402bdd16dfdd0887d6a1b327d9a908a2284d6221ecb24d2ab38f12d WatchSource:0}: Error finding container 10b2e6b07402bdd16dfdd0887d6a1b327d9a908a2284d6221ecb24d2ab38f12d: Status 404 returned error can't find the container with id 10b2e6b07402bdd16dfdd0887d6a1b327d9a908a2284d6221ecb24d2ab38f12d Oct 30 20:24:28.102776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:28.102683 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" event={"ID":"93386db5-1aeb-4d49-9efc-92113dc725ae","Type":"ContainerStarted","Data":"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c"} Oct 30 20:24:28.102776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:28.102737 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" event={"ID":"93386db5-1aeb-4d49-9efc-92113dc725ae","Type":"ContainerStarted","Data":"10b2e6b07402bdd16dfdd0887d6a1b327d9a908a2284d6221ecb24d2ab38f12d"} Oct 30 20:24:28.102999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:28.102783 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:28.152536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:28.152456 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" podStartSLOduration=2.152438151 podStartE2EDuration="2.152438151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:28.146766832 +0000 UTC m=+2992.276000754" watchObservedRunningTime="2026-10-30 20:24:28.152438151 +0000 UTC m=+2992.281672074" Oct 30 20:24:29.364355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:29.364317 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:30.679458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:30.679420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:30.848686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:30.848645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:32.095520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:32.095459 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:32.340221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:32.340173 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:33.071513 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.071470 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:33.332385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.332301 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:33.463296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.463258 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:33.748836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.748809 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 20:24:33.749114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.749082 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" podUID="55503847-60a8-4000-960c-8ea1e58fff3d" containerName="maas-api" containerID="cri-o://c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0" gracePeriod=30 Oct 30 20:24:33.775008 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:33.774974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:34.058440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.058411 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 20:24:34.148722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.148674 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\") pod \"55503847-60a8-4000-960c-8ea1e58fff3d\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " Oct 30 20:24:34.148923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.148752 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\" (UniqueName: \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\") pod \"55503847-60a8-4000-960c-8ea1e58fff3d\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " Oct 30 20:24:34.148923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.148900 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\") pod \"55503847-60a8-4000-960c-8ea1e58fff3d\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " Oct 30 20:24:34.149068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.148956 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") pod \"55503847-60a8-4000-960c-8ea1e58fff3d\" (UID: \"55503847-60a8-4000-960c-8ea1e58fff3d\") " Oct 30 20:24:34.149992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.149966 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca" pod "55503847-60a8-4000-960c-8ea1e58fff3d" (UID: "55503847-60a8-4000-960c-8ea1e58fff3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:34.154513 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.154042 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls" pod "55503847-60a8-4000-960c-8ea1e58fff3d" (UID: "55503847-60a8-4000-960c-8ea1e58fff3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.192095 4959 generic.go:356] "Generic (PLEG): container finished" podID="55503847-60a8-4000-960c-8ea1e58fff3d" containerID="c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0" exitCode=0 Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.192442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" event={"ID":"55503847-60a8-4000-960c-8ea1e58fff3d","Type":"ContainerDied","Data":"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0"} Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.192532 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" event={"ID":"55503847-60a8-4000-960c-8ea1e58fff3d","Type":"ContainerDied","Data":"6a5406657706aaba83a7fdec89806190e4cb024034a172f89e28637bb1e09599"} Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.192574 4959 scope.go:122] "RemoveContainer" containerID="c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0" Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.192825 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm" Oct 30 20:24:34.201881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.197176 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs" pod "55503847-60a8-4000-960c-8ea1e58fff3d" (UID: "55503847-60a8-4000-960c-8ea1e58fff3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:34.202681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.202648 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8" pod "55503847-60a8-4000-960c-8ea1e58fff3d" (UID: "55503847-60a8-4000-960c-8ea1e58fff3d"). InnerVolumeSpecName "kube-api-access-b7sg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:34.212143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.212121 4959 scope.go:122] "RemoveContainer" containerID="c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0" Oct 30 20:24:34.212556 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:34.212528 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0\": container with ID starting with c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0 not found: ID does not exist" containerID="c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0" Oct 30 20:24:34.212636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.212566 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0"} err="failed to get container status \"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0\": rpc error: code = NotFound desc = could not find container \"c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0\": container with ID starting with c927c9011197d825ae02ccc9fb22b7c5f97a0af833f54aa5699b802b40bd76b0 not found: ID does not exist" Oct 30 20:24:34.250511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.250459 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55503847-60a8-4000-960c-8ea1e58fff3d-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:34.250511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.250507 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:34.250511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.250520 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55503847-60a8-4000-960c-8ea1e58fff3d-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:34.250829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.250528 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7sg8\" (UniqueName: \"kubernetes.io/projected/55503847-60a8-4000-960c-8ea1e58fff3d-kube-api-access-b7sg8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:34.545751 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.545719 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 20:24:34.552099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.552072 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-043892-6555c44d4-2z9pm"] Oct 30 20:24:34.665595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.665552 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:34.812591 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.812476 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:34.930800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:34.930759 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:35.374579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:35.374538 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:36.001821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.001784 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55503847-60a8-4000-960c-8ea1e58fff3d" path="/var/lib/kubelet/pods/55503847-60a8-4000-960c-8ea1e58fff3d/volumes" Oct 30 20:24:36.473118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.473065 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:24:36.481210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.481183 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:24:36.690775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.690743 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:24:36.691874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.691849 4959 cpu_manager.go:414] "containerMap: removing container" podUID="55503847-60a8-4000-960c-8ea1e58fff3d" containerName="maas-api" Oct 30 20:24:36.691874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.691872 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55503847-60a8-4000-960c-8ea1e58fff3d" containerName="maas-api" Oct 30 20:24:36.692041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.692026 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="55503847-60a8-4000-960c-8ea1e58fff3d" containerName="maas-api" Oct 30 20:24:36.695888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.695867 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.718453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.718421 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f35cc1-serving-cert\"" Oct 30 20:24:36.736067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.735964 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:24:36.782533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.782484 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:36.785050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.785023 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt2n\" (UniqueName: \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.785141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.785093 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.785221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.785203 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.785261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.785251 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.891198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.891158 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.891427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.891269 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.891427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.891323 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.891577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.891530 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt2n\" (UniqueName: \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.892645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.892614 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.892802 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:36.892640 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-serving-cert: secret "maas-api-e2e-ipp-b-f35cc1-serving-cert" not found Oct 30 20:24:36.892802 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:36.892713 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls podName:345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d nodeName:}" failed. No retries permitted until 2026-10-30 20:24:37.392693268 +0000 UTC m=+3001.521927184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls") pod "maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" (UID: "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d") : secret "maas-api-e2e-ipp-b-f35cc1-serving-cert" not found Oct 30 20:24:36.908112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.908076 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:36.908585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.908550 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:36.911291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:36.911265 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt2n\" (UniqueName: \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:37.019075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:37.018979 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:37.398866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:37.398833 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:37.401512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:37.401465 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:37.610794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:37.610753 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:37.950729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:37.950697 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:24:37.952305 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:24:37.952272 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345f6b5a_1ce0_4e7b_8bf5_1eae3e6edd3d.slice/crio-0329dd3b3fa12ed3b4396bd939e7c74c96bebf8fac3cf5cf4bd7496ee2c2c152 WatchSource:0}: Error finding container 0329dd3b3fa12ed3b4396bd939e7c74c96bebf8fac3cf5cf4bd7496ee2c2c152: Status 404 returned error can't find the container with id 0329dd3b3fa12ed3b4396bd939e7c74c96bebf8fac3cf5cf4bd7496ee2c2c152 Oct 30 20:24:38.238374 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:38.238282 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" event={"ID":"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d","Type":"ContainerStarted","Data":"0329dd3b3fa12ed3b4396bd939e7c74c96bebf8fac3cf5cf4bd7496ee2c2c152"} Oct 30 20:24:38.728747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:38.728710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:39.115150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:39.115069 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:24:39.249056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:39.249017 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" event={"ID":"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d","Type":"ContainerStarted","Data":"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977"} Oct 30 20:24:39.249694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:39.249249 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:39.295465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:39.295407 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" podStartSLOduration=2.898476024 podStartE2EDuration="3.29539321s" totalImagesPullingTime="396.917186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:39.29374922 +0000 UTC m=+3003.422983140" watchObservedRunningTime="2026-10-30 20:24:39.29539321 +0000 UTC m=+3003.424627130" Oct 30 20:24:40.698438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:40.698403 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:40.906364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:40.906334 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 20:24:40.947144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:40.947119 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 20:24:40.993040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:40.992985 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\" (UniqueName: \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\") pod \"c8f83ae7-8073-4199-be91-5ff3531198d9\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " Oct 30 20:24:41.002946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.001638 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume" pod "c8f83ae7-8073-4199-be91-5ff3531198d9" (UID: "c8f83ae7-8073-4199-be91-5ff3531198d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:41.002946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.000822 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\") pod \"c8f83ae7-8073-4199-be91-5ff3531198d9\" (UID: \"c8f83ae7-8073-4199-be91-5ff3531198d9\") " Oct 30 20:24:41.002946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.002462 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8f83ae7-8073-4199-be91-5ff3531198d9-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:41.004786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.004755 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk" pod "c8f83ae7-8073-4199-be91-5ff3531198d9" (UID: "c8f83ae7-8073-4199-be91-5ff3531198d9"). InnerVolumeSpecName "kube-api-access-288nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:41.103378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.103295 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\" (UniqueName: \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\") pod \"7f7ecaa8-b517-4855-83ba-8e41ee262562\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " Oct 30 20:24:41.103595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.103489 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\") pod \"7f7ecaa8-b517-4855-83ba-8e41ee262562\" (UID: \"7f7ecaa8-b517-4855-83ba-8e41ee262562\") " Oct 30 20:24:41.103936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.103904 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume" pod "7f7ecaa8-b517-4855-83ba-8e41ee262562" (UID: "7f7ecaa8-b517-4855-83ba-8e41ee262562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:41.104199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.104183 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288nk\" (UniqueName: \"kubernetes.io/projected/c8f83ae7-8073-4199-be91-5ff3531198d9-kube-api-access-288nk\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:41.104246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.104205 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7ecaa8-b517-4855-83ba-8e41ee262562-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:41.105720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.105700 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd" pod "7f7ecaa8-b517-4855-83ba-8e41ee262562" (UID: "7f7ecaa8-b517-4855-83ba-8e41ee262562"). InnerVolumeSpecName "kube-api-access-mgmsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:41.205943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.205907 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgmsd\" (UniqueName: \"kubernetes.io/projected/7f7ecaa8-b517-4855-83ba-8e41ee262562-kube-api-access-mgmsd\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:41.263895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.263860 4959 generic.go:356] "Generic (PLEG): container finished" podID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerID="9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e" exitCode=0 Oct 30 20:24:41.264096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.263928 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" Oct 30 20:24:41.264096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.263942 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" event={"ID":"7f7ecaa8-b517-4855-83ba-8e41ee262562","Type":"ContainerDied","Data":"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e"} Oct 30 20:24:41.264096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.264001 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm" event={"ID":"7f7ecaa8-b517-4855-83ba-8e41ee262562","Type":"ContainerDied","Data":"e5d542d2fbc5c53d4aa0f7bb8d96cda521a3d7e0ab3b47b381f4ca3c1f6427e6"} Oct 30 20:24:41.264096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.264032 4959 scope.go:122] "RemoveContainer" containerID="9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e" Oct 30 20:24:41.266850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.266822 4959 generic.go:356] "Generic (PLEG): container finished" podID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerID="ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5" exitCode=0 Oct 30 20:24:41.266999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.266884 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" Oct 30 20:24:41.266999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.266913 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" event={"ID":"c8f83ae7-8073-4199-be91-5ff3531198d9","Type":"ContainerDied","Data":"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5"} Oct 30 20:24:41.266999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.266955 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb" event={"ID":"c8f83ae7-8073-4199-be91-5ff3531198d9","Type":"ContainerDied","Data":"2422f3312ea0c92ae22042da95a18cc9b7d0e7045f96290ef99ab1e4a912d951"} Oct 30 20:24:41.277784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.277560 4959 scope.go:122] "RemoveContainer" containerID="9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e" Oct 30 20:24:41.277958 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:41.277936 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e\": container with ID starting with 9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e not found: ID does not exist" containerID="9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e" Oct 30 20:24:41.278006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.277966 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e"} err="failed to get container status \"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e\": rpc error: code = NotFound desc = could not find container \"9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e\": container with ID starting with 9f90e57c1519e6aad8899fbc1291525d933fea8b59f261a7f400b82bf7a3493e not found: ID does not exist" Oct 30 20:24:41.278006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.277984 4959 scope.go:122] "RemoveContainer" containerID="ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5" Oct 30 20:24:41.288832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.288810 4959 scope.go:122] "RemoveContainer" containerID="ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5" Oct 30 20:24:41.289181 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:41.289152 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5\": container with ID starting with ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5 not found: ID does not exist" containerID="ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5" Oct 30 20:24:41.289256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.289191 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5"} err="failed to get container status \"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5\": rpc error: code = NotFound desc = could not find container \"ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5\": container with ID starting with ab80553e5ab7766e27efb7f8211339adc4f96586259ce7cec47d3c1db15cfbb5 not found: ID does not exist" Oct 30 20:24:41.317851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.317825 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 20:24:41.329058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.329029 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-043892-8d86f65cf-llvmm"] Oct 30 20:24:41.355107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.355007 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 20:24:41.358986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:41.358953 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-043892-68fdbd9748-b8hmb"] Oct 30 20:24:42.001256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.001224 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" path="/var/lib/kubelet/pods/7f7ecaa8-b517-4855-83ba-8e41ee262562/volumes" Oct 30 20:24:42.001806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.001789 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" path="/var/lib/kubelet/pods/c8f83ae7-8073-4199-be91-5ff3531198d9/volumes" Oct 30 20:24:42.044732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.044672 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:42.157832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.157799 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:42.387391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.387349 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 20:24:42.387828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.387744 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" containerID="cri-o://ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23" gracePeriod=30 Oct 30 20:24:42.415032 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.414982 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:42.534284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:42.534246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:44.616742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:44.616701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:45.261144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:45.261116 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:24:46.066930 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:46.066895 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:24:46.579711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:46.579669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:47.691392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.691364 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 20:24:47.799548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799483 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.799727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799598 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.799727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799666 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.799808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799739 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.799808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799798 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.799890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799826 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.799938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799927 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.800003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799986 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.800060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.799988 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.800130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800114 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.800195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800169 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\") pod \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\" (UID: \"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8\") " Oct 30 20:24:47.800257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800194 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.800257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800228 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:47.800478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800454 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.800839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800791 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.800839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800815 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.800839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800828 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.800839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800840 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.801302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.800854 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.802654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.802624 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "kube-api-access-9nfj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:47.802758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.802655 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:47.802946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.802926 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:47.803012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.802957 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy" pod "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" (UID: "bd1d9453-4769-4ab6-8b80-b767c3ba2bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:47.901987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.901953 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfj7\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-kube-api-access-9nfj7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.901987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.901983 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.901987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.901994 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:47.902220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:47.902004 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:48.321285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.321195 4959 generic.go:356] "Generic (PLEG): container finished" podID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerID="ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23" exitCode=0 Oct 30 20:24:48.321285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.321270 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" Oct 30 20:24:48.321485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.321287 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerDied","Data":"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23"} Oct 30 20:24:48.321485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.321349 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd" event={"ID":"bd1d9453-4769-4ab6-8b80-b767c3ba2bd8","Type":"ContainerDied","Data":"ba0826435f82d4585f5661e8970e40b5642e1a1a5804c05b204f0811e701a642"} Oct 30 20:24:48.321485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.321379 4959 scope.go:122] "RemoveContainer" containerID="ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23" Oct 30 20:24:48.334227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.334207 4959 scope.go:122] "RemoveContainer" containerID="17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d" Oct 30 20:24:48.344473 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.344441 4959 scope.go:122] "RemoveContainer" containerID="ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23" Oct 30 20:24:48.344788 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:48.344763 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23\": container with ID starting with ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23 not found: ID does not exist" containerID="ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23" Oct 30 20:24:48.344886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.344797 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23"} err="failed to get container status \"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23\": rpc error: code = NotFound desc = could not find container \"ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23\": container with ID starting with ee157066ea18f0eca595db193e16352c30d141f39772b4a12b8e49203f65af23 not found: ID does not exist" Oct 30 20:24:48.344886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.344816 4959 scope.go:122] "RemoveContainer" containerID="17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d" Oct 30 20:24:48.345098 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:48.345083 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d\": container with ID starting with 17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d not found: ID does not exist" containerID="17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d" Oct 30 20:24:48.345150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.345102 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d"} err="failed to get container status \"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d\": rpc error: code = NotFound desc = could not find container \"17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d\": container with ID starting with 17133ad98d2e8ab6f5e8e55ff120b85e98f5426d305f4954304ee58cdbe1b21d not found: ID does not exist" Oct 30 20:24:48.383264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.383232 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 20:24:48.394174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.394149 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-043892-openshift-default-8457464759-5t5bd"] Oct 30 20:24:48.659214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.659177 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:48.707709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.707607 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 20:24:48.710748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:48.709612 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" containerID="cri-o://929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4" gracePeriod=30 Oct 30 20:24:50.000580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:50.000546 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" path="/var/lib/kubelet/pods/bd1d9453-4769-4ab6-8b80-b767c3ba2bd8/volumes" Oct 30 20:24:50.482935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:50.482904 4959 patch_prober.go:29] interesting pod/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:24:50.483110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:50.482955 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:24:50.726530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:50.726466 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:50.837126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:50.837047 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:51.135559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:51.135532 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:52.925827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:52.925763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:53.130925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:53.130895 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 20:24:53.165706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:53.165672 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 20:24:53.276911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:53.276813 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:53.402742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:53.402710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.013218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.013192 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 20:24:54.095333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095302 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095336 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095377 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095449 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095598 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095637 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:54.095813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095690 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095734 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095854 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.095996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095859 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:54.095996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.095956 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\") pod \"dd37d968-606c-4651-8bb8-fbc56826f361\" (UID: \"dd37d968-606c-4651-8bb8-fbc56826f361\") " Oct 30 20:24:54.096537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.096480 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:54.097485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.097068 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:54.097759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.097680 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:54.098828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.098361 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:54.098828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.098764 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.098828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.098785 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.098828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.098799 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd37d968-606c-4651-8bb8-fbc56826f361-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.099439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.099413 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:54.100553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.100527 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "kube-api-access-plhbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:54.101195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.101164 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo" pod "dd37d968-606c-4651-8bb8-fbc56826f361" (UID: "dd37d968-606c-4651-8bb8-fbc56826f361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:54.200269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200224 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.200269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200263 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.200269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200277 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.200641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200292 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhbz\" (UniqueName: \"kubernetes.io/projected/dd37d968-606c-4651-8bb8-fbc56826f361-kube-api-access-plhbz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.200641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200307 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd37d968-606c-4651-8bb8-fbc56826f361-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.200641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.200325 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd37d968-606c-4651-8bb8-fbc56826f361-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:54.370923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.370827 4959 generic.go:356] "Generic (PLEG): container finished" podID="dd37d968-606c-4651-8bb8-fbc56826f361" containerID="929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4" exitCode=0 Oct 30 20:24:54.370923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.370902 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" Oct 30 20:24:54.370923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.370914 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" event={"ID":"dd37d968-606c-4651-8bb8-fbc56826f361","Type":"ContainerDied","Data":"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4"} Oct 30 20:24:54.371233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.370981 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2" event={"ID":"dd37d968-606c-4651-8bb8-fbc56826f361","Type":"ContainerDied","Data":"225c4274aa0629fbd6a606d4e62cb7dff327df7f2b6ba2360161e888bdce2170"} Oct 30 20:24:54.371233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.371008 4959 scope.go:122] "RemoveContainer" containerID="929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4" Oct 30 20:24:54.388352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.388324 4959 scope.go:122] "RemoveContainer" containerID="929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4" Oct 30 20:24:54.388691 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:54.388671 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4\": container with ID starting with 929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4 not found: ID does not exist" containerID="929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4" Oct 30 20:24:54.388783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.388700 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4"} err="failed to get container status \"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4\": rpc error: code = NotFound desc = could not find container \"929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4\": container with ID starting with 929ea6d2f12d90cde7426605411bae9ffb135952614c9ff83dd010226cc557b4 not found: ID does not exist" Oct 30 20:24:54.426053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.426022 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 20:24:54.430190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.430166 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6c4dd1-openshift-default-8758b795f-94cl2"] Oct 30 20:24:54.875521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:54.875460 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:56.003145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:56.003106 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" path="/var/lib/kubelet/pods/dd37d968-606c-4651-8bb8-fbc56826f361/volumes" Oct 30 20:24:56.606545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:56.606510 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 20:24:56.606757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:56.606731 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" podUID="13faa547-2544-4c5c-8499-4efe064e86af" containerName="maas-api" containerID="cri-o://5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1" gracePeriod=30 Oct 30 20:24:56.672721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:56.672686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:56.882981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:56.882954 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 20:24:57.016447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.016401 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:24:57.016919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.016633 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" podUID="71e114e5-725a-44fe-b8ef-4edabae56e2d" containerName="maas-api" containerID="cri-o://477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9" gracePeriod=30 Oct 30 20:24:57.044110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.044078 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\") pod \"13faa547-2544-4c5c-8499-4efe064e86af\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " Oct 30 20:24:57.044250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.044143 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\") pod \"13faa547-2544-4c5c-8499-4efe064e86af\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " Oct 30 20:24:57.044438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.044420 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") pod \"13faa547-2544-4c5c-8499-4efe064e86af\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " Oct 30 20:24:57.044478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.044460 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\" (UniqueName: \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\") pod \"13faa547-2544-4c5c-8499-4efe064e86af\" (UID: \"13faa547-2544-4c5c-8499-4efe064e86af\") " Oct 30 20:24:57.046256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.046224 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca" pod "13faa547-2544-4c5c-8499-4efe064e86af" (UID: "13faa547-2544-4c5c-8499-4efe064e86af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:57.047855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.047814 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc" pod "13faa547-2544-4c5c-8499-4efe064e86af" (UID: "13faa547-2544-4c5c-8499-4efe064e86af"). InnerVolumeSpecName "kube-api-access-dw6lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:57.049042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.049018 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls" pod "13faa547-2544-4c5c-8499-4efe064e86af" (UID: "13faa547-2544-4c5c-8499-4efe064e86af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:57.050466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.050441 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs" pod "13faa547-2544-4c5c-8499-4efe064e86af" (UID: "13faa547-2544-4c5c-8499-4efe064e86af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:57.091174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.091143 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:57.146808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.146780 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.146808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.146808 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6lc\" (UniqueName: \"kubernetes.io/projected/13faa547-2544-4c5c-8499-4efe064e86af-kube-api-access-dw6lc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.147061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.146821 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13faa547-2544-4c5c-8499-4efe064e86af-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.147061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.146834 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13faa547-2544-4c5c-8499-4efe064e86af-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.282483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.282448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:57.313750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.313725 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:24:57.393427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.393340 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:57.397422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.397395 4959 generic.go:356] "Generic (PLEG): container finished" podID="71e114e5-725a-44fe-b8ef-4edabae56e2d" containerID="477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9" exitCode=0 Oct 30 20:24:57.397658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.397447 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" Oct 30 20:24:57.397658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.397459 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" event={"ID":"71e114e5-725a-44fe-b8ef-4edabae56e2d","Type":"ContainerDied","Data":"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9"} Oct 30 20:24:57.397658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.397601 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb" event={"ID":"71e114e5-725a-44fe-b8ef-4edabae56e2d","Type":"ContainerDied","Data":"c5a6efdaf9db134873f4be20102f4b1ac14bea5db84ca76c32107f928681ee1b"} Oct 30 20:24:57.397658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.397637 4959 scope.go:122] "RemoveContainer" containerID="477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9" Oct 30 20:24:57.399615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.399589 4959 generic.go:356] "Generic (PLEG): container finished" podID="13faa547-2544-4c5c-8499-4efe064e86af" containerID="5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1" exitCode=0 Oct 30 20:24:57.399773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.399747 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" event={"ID":"13faa547-2544-4c5c-8499-4efe064e86af","Type":"ContainerDied","Data":"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1"} Oct 30 20:24:57.399867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.399799 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" event={"ID":"13faa547-2544-4c5c-8499-4efe064e86af","Type":"ContainerDied","Data":"fc444e8c83e890cd1f5764497e64034d4a8bcb10bd098c203f2efb594fba7f8b"} Oct 30 20:24:57.400012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.399988 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz" Oct 30 20:24:57.410380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.410358 4959 scope.go:122] "RemoveContainer" containerID="477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9" Oct 30 20:24:57.410686 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:57.410666 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9\": container with ID starting with 477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9 not found: ID does not exist" containerID="477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9" Oct 30 20:24:57.410768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.410692 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9"} err="failed to get container status \"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9\": rpc error: code = NotFound desc = could not find container \"477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9\": container with ID starting with 477d5b2185eae7968021293760b4c5940c2ea2d666035c716d40ddd83b1563a9 not found: ID does not exist" Oct 30 20:24:57.410768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.410715 4959 scope.go:122] "RemoveContainer" containerID="5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1" Oct 30 20:24:57.421337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.421322 4959 scope.go:122] "RemoveContainer" containerID="5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1" Oct 30 20:24:57.421683 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:24:57.421665 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1\": container with ID starting with 5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1 not found: ID does not exist" containerID="5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1" Oct 30 20:24:57.421748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.421689 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1"} err="failed to get container status \"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1\": rpc error: code = NotFound desc = could not find container \"5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1\": container with ID starting with 5571c7bfbc9a22f4a6e78e35e130da3ec141e1e03d1cfbf901b1d2bbb492fff1 not found: ID does not exist" Oct 30 20:24:57.452703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.452680 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\" (UniqueName: \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\") pod \"71e114e5-725a-44fe-b8ef-4edabae56e2d\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " Oct 30 20:24:57.452950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.452933 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\") pod \"71e114e5-725a-44fe-b8ef-4edabae56e2d\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " Oct 30 20:24:57.453028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.453008 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\") pod \"71e114e5-725a-44fe-b8ef-4edabae56e2d\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " Oct 30 20:24:57.453091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.453055 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\") pod \"71e114e5-725a-44fe-b8ef-4edabae56e2d\" (UID: \"71e114e5-725a-44fe-b8ef-4edabae56e2d\") " Oct 30 20:24:57.453448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.453426 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca" pod "71e114e5-725a-44fe-b8ef-4edabae56e2d" (UID: "71e114e5-725a-44fe-b8ef-4edabae56e2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:57.453641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.453625 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e114e5-725a-44fe-b8ef-4edabae56e2d-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.455199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.455176 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls" pod "71e114e5-725a-44fe-b8ef-4edabae56e2d" (UID: "71e114e5-725a-44fe-b8ef-4edabae56e2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:57.455578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.455556 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs" pod "71e114e5-725a-44fe-b8ef-4edabae56e2d" (UID: "71e114e5-725a-44fe-b8ef-4edabae56e2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:24:57.455687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.455558 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw" pod "71e114e5-725a-44fe-b8ef-4edabae56e2d" (UID: "71e114e5-725a-44fe-b8ef-4edabae56e2d"). InnerVolumeSpecName "kube-api-access-bzvsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:57.465383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.465355 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 20:24:57.468435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.468414 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-159b3a-768fddc4b4-twknz"] Oct 30 20:24:57.509198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.509171 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:57.554643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.554604 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.554643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.554639 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71e114e5-725a-44fe-b8ef-4edabae56e2d-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.554857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.554668 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvsw\" (UniqueName: \"kubernetes.io/projected/71e114e5-725a-44fe-b8ef-4edabae56e2d-kube-api-access-bzvsw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:24:57.783158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.783118 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:24:57.787645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.787619 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6175d5-5758b95694-4m4wb"] Oct 30 20:24:57.926902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:57.926868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:58.000106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.000016 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13faa547-2544-4c5c-8499-4efe064e86af" path="/var/lib/kubelet/pods/13faa547-2544-4c5c-8499-4efe064e86af/volumes" Oct 30 20:24:58.000470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.000456 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e114e5-725a-44fe-b8ef-4edabae56e2d" path="/var/lib/kubelet/pods/71e114e5-725a-44fe-b8ef-4edabae56e2d/volumes" Oct 30 20:24:58.132235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.132200 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" containerID="cri-o://e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab" gracePeriod=25 Oct 30 20:24:58.166579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.166549 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" containerID="cri-o://f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55" gracePeriod=25 Oct 30 20:24:58.189757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.189727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:58.336109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.335896 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:58.445774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.445732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:58.851825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:24:58.851790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:00.753116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:00.753079 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:01.279764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:01.279718 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:01.279939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:01.279864 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:02.532334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:02.532298 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:04.262172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:04.262135 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:05.290811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.290768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:05.399799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.399751 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:05.633308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.633277 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:25:05.633542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.633513 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" containerID="cri-o://7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41" gracePeriod=30 Oct 30 20:25:05.667917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.667881 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:05.784137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:05.784103 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:06.102765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:06.102680 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:06.247324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:06.247285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:06.545917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:06.545882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:09.272119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:09.272063 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:09.978956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:09.978925 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:10.091605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:10.091572 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:10.911089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:10.911032 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:25:11.050860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.050825 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.050860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.050880 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.050927 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.050965 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051010 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051082 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051128 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051172 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051245 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\") pod \"dbafd718-e34e-40fb-922e-b648acfff217\" (UID: \"dbafd718-e34e-40fb-922e-b648acfff217\") " Oct 30 20:25:11.051638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051574 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.051638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.051592 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.052068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052002 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.052187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052109 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.052187 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052122 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:11.052553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052520 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbafd718-e34e-40fb-922e-b648acfff217-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.052553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052549 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.052769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052563 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.052769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052576 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.052769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.052592 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.053951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.053921 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:25:11.054144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.054126 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:11.054572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.054551 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "kube-api-access-ft992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:11.054644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.054569 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy" pod "dbafd718-e34e-40fb-922e-b648acfff217" (UID: "dbafd718-e34e-40fb-922e-b648acfff217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:11.114407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.114366 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 20:25:11.153641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.153597 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft992\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-kube-api-access-ft992\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.153641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.153641 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbafd718-e34e-40fb-922e-b648acfff217-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.153860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.153656 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbafd718-e34e-40fb-922e-b648acfff217-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.153860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.153671 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbafd718-e34e-40fb-922e-b648acfff217-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:11.234236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.234153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:11.280467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.280417 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:11.280690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.280441 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:25:11.343917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.343861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:11.498818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.498726 4959 generic.go:356] "Generic (PLEG): container finished" podID="dbafd718-e34e-40fb-922e-b648acfff217" containerID="7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41" exitCode=0 Oct 30 20:25:11.498818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.498775 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" Oct 30 20:25:11.499019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.498820 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" event={"ID":"dbafd718-e34e-40fb-922e-b648acfff217","Type":"ContainerDied","Data":"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41"} Oct 30 20:25:11.499019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.498876 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc" event={"ID":"dbafd718-e34e-40fb-922e-b648acfff217","Type":"ContainerDied","Data":"47e03676b19f4f910f47f3562d1619d2f013e266acb9942f10f40abd19b41364"} Oct 30 20:25:11.499019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.498898 4959 scope.go:122] "RemoveContainer" containerID="7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41" Oct 30 20:25:11.511012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.510996 4959 scope.go:122] "RemoveContainer" containerID="7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41" Oct 30 20:25:11.511299 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:11.511282 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41\": container with ID starting with 7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41 not found: ID does not exist" containerID="7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41" Oct 30 20:25:11.511344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.511306 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41"} err="failed to get container status \"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41\": rpc error: code = NotFound desc = could not find container \"7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41\": container with ID starting with 7e4d3e96f770adba7fb95979f49c677343c48fd2ade544899fbe3374865bab41 not found: ID does not exist" Oct 30 20:25:11.552941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.552908 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:25:11.558444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:11.558412 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-159b3a-openshift-default-85765b89f-x6zzc"] Oct 30 20:25:12.003521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:12.003466 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbafd718-e34e-40fb-922e-b648acfff217" path="/var/lib/kubelet/pods/dbafd718-e34e-40fb-922e-b648acfff217/volumes" Oct 30 20:25:13.105920 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:13.105891 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:14.760023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.759992 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:25:14.760688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760672 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:14.760745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760689 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:14.760745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760710 4959 cpu_manager.go:414] "containerMap: removing container" podUID="13faa547-2544-4c5c-8499-4efe064e86af" containerName="maas-api" Oct 30 20:25:14.760745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760714 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13faa547-2544-4c5c-8499-4efe064e86af" containerName="maas-api" Oct 30 20:25:14.760745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760723 4959 cpu_manager.go:414] "containerMap: removing container" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" Oct 30 20:25:14.760745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760731 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760749 4959 cpu_manager.go:414] "containerMap: removing container" podUID="71e114e5-725a-44fe-b8ef-4edabae56e2d" containerName="maas-api" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760756 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e114e5-725a-44fe-b8ef-4edabae56e2d" containerName="maas-api" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760774 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760781 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760790 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760796 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760825 4959 cpu_manager.go:414] "containerMap: removing container" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" Oct 30 20:25:14.760959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.760833 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761012 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761027 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbafd718-e34e-40fb-922e-b648acfff217" containerName="istio-proxy" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761044 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761057 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="13faa547-2544-4c5c-8499-4efe064e86af" containerName="maas-api" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761068 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd37d968-606c-4651-8bb8-fbc56826f361" containerName="istio-proxy" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761079 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e114e5-725a-44fe-b8ef-4edabae56e2d" containerName="maas-api" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761087 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7ecaa8-b517-4855-83ba-8e41ee262562" containerName="payload-pre-processing" Oct 30 20:25:14.761212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.761097 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f83ae7-8073-4199-be91-5ff3531198d9" containerName="payload-processing" Oct 30 20:25:14.767131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.767109 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:14.784046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.784024 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c19563-serving-cert\"" Oct 30 20:25:14.801076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.801047 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:25:14.860709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.860679 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:14.901803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.901777 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4b7\" (UniqueName: \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:14.901924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.901822 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:14.901924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.901896 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:14.902072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.902053 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:14.973542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:14.973488 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:15.004527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.004467 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4b7\" (UniqueName: \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.004706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.004548 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.004706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.004630 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.004826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.004789 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.004948 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:15.004915 4959 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-serving-cert: secret "maas-api-e2e-ipp-a-c19563-serving-cert" not found Oct 30 20:25:15.005030 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:15.004996 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls podName:cda9f9c3-c416-471b-9dcb-76bac5837aaf nodeName:}" failed. No retries permitted until 2026-10-30 20:25:15.504977018 +0000 UTC m=+3039.634210944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls") pod "maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" (UID: "cda9f9c3-c416-471b-9dcb-76bac5837aaf") : secret "maas-api-e2e-ipp-a-c19563-serving-cert" not found Oct 30 20:25:15.005356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.005334 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.007481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.007452 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.026441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.026379 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4b7\" (UniqueName: \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.348001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.347921 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:15.511907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.511867 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.514955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.514924 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:15.689180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:15.689142 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:16.014254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.014229 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:25:16.073015 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:16.072985 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:25:16.539815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.539782 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" event={"ID":"cda9f9c3-c416-471b-9dcb-76bac5837aaf","Type":"ContainerStarted","Data":"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847"} Oct 30 20:25:16.539995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.539835 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" event={"ID":"cda9f9c3-c416-471b-9dcb-76bac5837aaf","Type":"ContainerStarted","Data":"002cf805239e7f6a3b11c4420d28846f5ea97275756e2688ab6656732d3f2895"} Oct 30 20:25:16.540071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.539999 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:16.586910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.586848 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" podStartSLOduration=2.182936301 podStartE2EDuration="2.586828966s" totalImagesPullingTime="403.892665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:16.582735521 +0000 UTC m=+3040.711969480" watchObservedRunningTime="2026-10-30 20:25:16.586828966 +0000 UTC m=+3040.716062896" Oct 30 20:25:16.834115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:16.834027 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:17.299667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:17.299621 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:17.411578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:17.411545 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:18.457478 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.457452 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 20:25:18.463994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.463968 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\" (UniqueName: \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\") pod \"43d243ba-2779-4a6c-abd0-498701701489\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " Oct 30 20:25:18.464115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.464046 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\") pod \"43d243ba-2779-4a6c-abd0-498701701489\" (UID: \"43d243ba-2779-4a6c-abd0-498701701489\") " Oct 30 20:25:18.464409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.464389 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume" pod "43d243ba-2779-4a6c-abd0-498701701489" (UID: "43d243ba-2779-4a6c-abd0-498701701489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:18.464635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.464615 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d243ba-2779-4a6c-abd0-498701701489-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:18.466261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.466241 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq" pod "43d243ba-2779-4a6c-abd0-498701701489" (UID: "43d243ba-2779-4a6c-abd0-498701701489"). InnerVolumeSpecName "kube-api-access-6lngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:18.483467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.483449 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 20:25:18.561962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.561921 4959 generic.go:356] "Generic (PLEG): container finished" podID="43d243ba-2779-4a6c-abd0-498701701489" containerID="e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab" exitCode=0 Oct 30 20:25:18.562199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.562135 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" event={"ID":"43d243ba-2779-4a6c-abd0-498701701489","Type":"ContainerDied","Data":"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab"} Oct 30 20:25:18.562199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.562188 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" event={"ID":"43d243ba-2779-4a6c-abd0-498701701489","Type":"ContainerDied","Data":"cd6cf74a96d4e0669d46da199f95ecf29f0dad9e070472ec23ec56c71f0bfb3e"} Oct 30 20:25:18.562354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.562216 4959 scope.go:122] "RemoveContainer" containerID="e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab" Oct 30 20:25:18.562354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.562216 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch" Oct 30 20:25:18.564699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.564680 4959 generic.go:356] "Generic (PLEG): container finished" podID="333d21e4-88f9-4bab-997f-2c82518767a2" containerID="f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55" exitCode=0 Oct 30 20:25:18.564960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.564850 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" event={"ID":"333d21e4-88f9-4bab-997f-2c82518767a2","Type":"ContainerDied","Data":"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55"} Oct 30 20:25:18.565097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.565084 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" event={"ID":"333d21e4-88f9-4bab-997f-2c82518767a2","Type":"ContainerDied","Data":"ed07ff725d01d14e750a29478090a2fa796743ff8d9c9be5430fdcfbbbbe6d22"} Oct 30 20:25:18.565188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.564862 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm" Oct 30 20:25:18.565351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.565323 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\") pod \"333d21e4-88f9-4bab-997f-2c82518767a2\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " Oct 30 20:25:18.565421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.565386 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\" (UniqueName: \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\") pod \"333d21e4-88f9-4bab-997f-2c82518767a2\" (UID: \"333d21e4-88f9-4bab-997f-2c82518767a2\") " Oct 30 20:25:18.565915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.565889 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lngq\" (UniqueName: \"kubernetes.io/projected/43d243ba-2779-4a6c-abd0-498701701489-kube-api-access-6lngq\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:18.565915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.565888 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume" pod "333d21e4-88f9-4bab-997f-2c82518767a2" (UID: "333d21e4-88f9-4bab-997f-2c82518767a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:18.568759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.568723 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr" pod "333d21e4-88f9-4bab-997f-2c82518767a2" (UID: "333d21e4-88f9-4bab-997f-2c82518767a2"). InnerVolumeSpecName "kube-api-access-wvsnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:18.610219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.610199 4959 scope.go:122] "RemoveContainer" containerID="e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab" Oct 30 20:25:18.610576 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:18.610555 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab\": container with ID starting with e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab not found: ID does not exist" containerID="e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab" Oct 30 20:25:18.610635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.610585 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab"} err="failed to get container status \"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab\": rpc error: code = NotFound desc = could not find container \"e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab\": container with ID starting with e38f50d63d7624dcccf69561090c250a1b0eee520326585c6d7c46b0a1afbaab not found: ID does not exist" Oct 30 20:25:18.610635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.610602 4959 scope.go:122] "RemoveContainer" containerID="f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55" Oct 30 20:25:18.615301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.615275 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 20:25:18.619620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.619597 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-159b3a-69956d5dcb-6fjch"] Oct 30 20:25:18.621941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.621925 4959 scope.go:122] "RemoveContainer" containerID="f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55" Oct 30 20:25:18.622194 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:18.622177 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55\": container with ID starting with f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55 not found: ID does not exist" containerID="f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55" Oct 30 20:25:18.622244 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.622203 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55"} err="failed to get container status \"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55\": rpc error: code = NotFound desc = could not find container \"f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55\": container with ID starting with f90f44a54390e735f63c67ce350bc941f054ff9000d62b009bec69f005497c55 not found: ID does not exist" Oct 30 20:25:18.667930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.667853 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/333d21e4-88f9-4bab-997f-2c82518767a2-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:18.667930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.667887 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsnr\" (UniqueName: \"kubernetes.io/projected/333d21e4-88f9-4bab-997f-2c82518767a2-kube-api-access-wvsnr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:18.759268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.759235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:18.915364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.915333 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 20:25:18.923012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:18.922947 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-159b3a-7b6dbcc77f-47lfm"] Oct 30 20:25:19.997850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:19.997819 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" path="/var/lib/kubelet/pods/333d21e4-88f9-4bab-997f-2c82518767a2/volumes" Oct 30 20:25:19.998300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:19.998285 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d243ba-2779-4a6c-abd0-498701701489" path="/var/lib/kubelet/pods/43d243ba-2779-4a6c-abd0-498701701489/volumes" Oct 30 20:25:20.746295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:20.746263 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:25:20.746565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:20.746522 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" podUID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" containerName="maas-api" containerID="cri-o://d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a" gracePeriod=30 Oct 30 20:25:20.819082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:20.819053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:21.024302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.024261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:21.058279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.058253 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:25:21.098444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.098418 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\") pod \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " Oct 30 20:25:21.098654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.098482 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") pod \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " Oct 30 20:25:21.098654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.098607 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\") pod \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " Oct 30 20:25:21.098812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.098780 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca" pod "fee95dd6-e2a1-437d-a529-d1a50e4096ad" (UID: "fee95dd6-e2a1-437d-a529-d1a50e4096ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:21.098946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.098823 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\" (UniqueName: \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\") pod \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\" (UID: \"fee95dd6-e2a1-437d-a529-d1a50e4096ad\") " Oct 30 20:25:21.099422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.099398 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee95dd6-e2a1-437d-a529-d1a50e4096ad-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:21.101468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.101436 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs" pod "fee95dd6-e2a1-437d-a529-d1a50e4096ad" (UID: "fee95dd6-e2a1-437d-a529-d1a50e4096ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:21.101696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.101670 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls" pod "fee95dd6-e2a1-437d-a529-d1a50e4096ad" (UID: "fee95dd6-e2a1-437d-a529-d1a50e4096ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:21.101767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.101755 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg" pod "fee95dd6-e2a1-437d-a529-d1a50e4096ad" (UID: "fee95dd6-e2a1-437d-a529-d1a50e4096ad"). InnerVolumeSpecName "kube-api-access-h8cdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:21.147799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.147768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:21.200792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.200760 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:21.200792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.200788 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee95dd6-e2a1-437d-a529-d1a50e4096ad-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:21.200792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.200799 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cdg\" (UniqueName: \"kubernetes.io/projected/fee95dd6-e2a1-437d-a529-d1a50e4096ad-kube-api-access-h8cdg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:21.592220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.592178 4959 generic.go:356] "Generic (PLEG): container finished" podID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" containerID="d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a" exitCode=0 Oct 30 20:25:21.592409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.592238 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" Oct 30 20:25:21.592409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.592236 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" event={"ID":"fee95dd6-e2a1-437d-a529-d1a50e4096ad","Type":"ContainerDied","Data":"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a"} Oct 30 20:25:21.592409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.592310 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5" event={"ID":"fee95dd6-e2a1-437d-a529-d1a50e4096ad","Type":"ContainerDied","Data":"38bfef699a70ed2a154ef85c82a7d9e917009def798436acf4f6396d49f748d7"} Oct 30 20:25:21.592409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.592343 4959 scope.go:122] "RemoveContainer" containerID="d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a" Oct 30 20:25:21.603913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.603894 4959 scope.go:122] "RemoveContainer" containerID="d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a" Oct 30 20:25:21.604230 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:21.604208 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a\": container with ID starting with d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a not found: ID does not exist" containerID="d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a" Oct 30 20:25:21.604336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.604236 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a"} err="failed to get container status \"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a\": rpc error: code = NotFound desc = could not find container \"d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a\": container with ID starting with d44da6a0174fa8993fc1b09e193aba0cb3c3a325f83d3f508f0a4183baba068a not found: ID does not exist" Oct 30 20:25:21.662549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.662521 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:25:21.673093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.673070 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7eec8-5697bfcfbf-vm2w5"] Oct 30 20:25:21.686822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.686786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:21.780614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.780581 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:25:21.816775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.816736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:21.933761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:21.933727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:22.013517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:22.011781 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" path="/var/lib/kubelet/pods/fee95dd6-e2a1-437d-a529-d1a50e4096ad/volumes" Oct 30 20:25:22.416559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:22.416520 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:22.554803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:22.554774 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:25:23.868220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:23.868191 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:25.621066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:25.621021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:26.137780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.137710 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" containerID="cri-o://6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc" gracePeriod=45 Oct 30 20:25:26.149081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.149047 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": dial tcp 10.132.0.70:8000: connect: connection refused" Oct 30 20:25:26.441201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.441174 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 20:25:26.480544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480480 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.480760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480580 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.480760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480645 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.480760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480720 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.480938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480855 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\" (UniqueName: \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.480938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480870 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.480938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480883 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.480938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480907 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\" (UniqueName: \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\") pod \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\" (UID: \"60e5ea21-98c1-4a84-9532-da59c98c4fe2\") " Oct 30 20:25:26.481139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.480984 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.481418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.481400 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.481532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.481424 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.481532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.481436 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.483203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.483177 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:26.483300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.483215 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "kube-api-access-pnm2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:26.483300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.483221 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs" pod "60e5ea21-98c1-4a84-9532-da59c98c4fe2" (UID: "60e5ea21-98c1-4a84-9532-da59c98c4fe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:26.583147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.583110 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnm2f\" (UniqueName: \"kubernetes.io/projected/60e5ea21-98c1-4a84-9532-da59c98c4fe2-kube-api-access-pnm2f\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.583147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.583146 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60e5ea21-98c1-4a84-9532-da59c98c4fe2-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.583424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.583160 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60e5ea21-98c1-4a84-9532-da59c98c4fe2-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:26.628867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.628833 4959 generic.go:356] "Generic (PLEG): container finished" podID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerID="6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc" exitCode=0 Oct 30 20:25:26.629313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.628877 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" Oct 30 20:25:26.629313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.628918 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" event={"ID":"60e5ea21-98c1-4a84-9532-da59c98c4fe2","Type":"ContainerDied","Data":"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc"} Oct 30 20:25:26.629313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.628959 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5" event={"ID":"60e5ea21-98c1-4a84-9532-da59c98c4fe2","Type":"ContainerDied","Data":"ccb52fe1c0d03cee504c59f8660b306cccdb544c337c59b60122e85dc88d4119"} Oct 30 20:25:26.629313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.628976 4959 scope.go:122] "RemoveContainer" containerID="6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc" Oct 30 20:25:26.640302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.640281 4959 scope.go:122] "RemoveContainer" containerID="6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc" Oct 30 20:25:26.640574 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:26.640553 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc\": container with ID starting with 6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc not found: ID does not exist" containerID="6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc" Oct 30 20:25:26.640647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.640581 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc"} err="failed to get container status \"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc\": rpc error: code = NotFound desc = could not find container \"6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc\": container with ID starting with 6cbc5429131d21877e316b83a76b0b13ab83d1c9752d7fe9485b0b6ab40676bc not found: ID does not exist" Oct 30 20:25:26.682973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.682871 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 20:25:26.687131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:26.687099 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a3426/distinct-2-w4-8a3426-kserve-5784789f98-nb6w5"] Oct 30 20:25:27.574410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:27.574376 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:27.999793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:27.999759 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" path="/var/lib/kubelet/pods/60e5ea21-98c1-4a84-9532-da59c98c4fe2/volumes" Oct 30 20:25:28.821247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:28.821211 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:25:28.858917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:28.858884 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:25:28.917562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:28.917487 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.034010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.033977 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.165708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.165681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.273036 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.273003 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.444775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.444701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.734626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.734546 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:29.941096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:29.941061 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:25:30.001828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:30.001743 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:31.691487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:31.691455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:33.202390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:33.202361 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:33.638284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:33.638255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:33.822720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:33.822669 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" podUID="7291d7dd-9676-440a-9b38-99eb02326cd1" containerName="payload-processing" containerID="cri-o://f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d" gracePeriod=25 Oct 30 20:25:33.859645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:33.859616 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" podUID="fc92b9a6-e284-450a-becd-e88ba64484dc" containerName="payload-pre-processing" containerID="cri-o://ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23" gracePeriod=25 Oct 30 20:25:34.178424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.178403 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:25:34.182020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.181995 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:25:34.304458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.304428 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\" (UniqueName: \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\") pod \"7291d7dd-9676-440a-9b38-99eb02326cd1\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " Oct 30 20:25:34.304903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.304468 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\") pod \"7291d7dd-9676-440a-9b38-99eb02326cd1\" (UID: \"7291d7dd-9676-440a-9b38-99eb02326cd1\") " Oct 30 20:25:34.304903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.304628 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\" (UniqueName: \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\") pod \"fc92b9a6-e284-450a-becd-e88ba64484dc\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " Oct 30 20:25:34.304903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.304767 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\") pod \"fc92b9a6-e284-450a-becd-e88ba64484dc\" (UID: \"fc92b9a6-e284-450a-becd-e88ba64484dc\") " Oct 30 20:25:34.305103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.305081 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume" pod "7291d7dd-9676-440a-9b38-99eb02326cd1" (UID: "7291d7dd-9676-440a-9b38-99eb02326cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.305155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.305129 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume" pod "fc92b9a6-e284-450a-becd-e88ba64484dc" (UID: "fc92b9a6-e284-450a-becd-e88ba64484dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:34.305648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.305625 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7291d7dd-9676-440a-9b38-99eb02326cd1-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.305726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.305655 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92b9a6-e284-450a-becd-e88ba64484dc-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.306951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.306926 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h" pod "fc92b9a6-e284-450a-becd-e88ba64484dc" (UID: "fc92b9a6-e284-450a-becd-e88ba64484dc"). InnerVolumeSpecName "kube-api-access-9cf2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.307069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.307027 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl" pod "7291d7dd-9676-440a-9b38-99eb02326cd1" (UID: "7291d7dd-9676-440a-9b38-99eb02326cd1"). InnerVolumeSpecName "kube-api-access-wm4tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:34.406783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.406750 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm4tl\" (UniqueName: \"kubernetes.io/projected/7291d7dd-9676-440a-9b38-99eb02326cd1-kube-api-access-wm4tl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.406783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.406782 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf2h\" (UniqueName: \"kubernetes.io/projected/fc92b9a6-e284-450a-becd-e88ba64484dc-kube-api-access-9cf2h\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:34.714189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.714158 4959 generic.go:356] "Generic (PLEG): container finished" podID="fc92b9a6-e284-450a-becd-e88ba64484dc" containerID="ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23" exitCode=0 Oct 30 20:25:34.714402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.714216 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" Oct 30 20:25:34.714402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.714254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" event={"ID":"fc92b9a6-e284-450a-becd-e88ba64484dc","Type":"ContainerDied","Data":"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23"} Oct 30 20:25:34.714402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.714344 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27" event={"ID":"fc92b9a6-e284-450a-becd-e88ba64484dc","Type":"ContainerDied","Data":"f32298ced86996a6a7d599c140f938806b3ef679c47013ddf7e765df94f35fa7"} Oct 30 20:25:34.714402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.714373 4959 scope.go:122] "RemoveContainer" containerID="ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23" Oct 30 20:25:34.715767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.715736 4959 generic.go:356] "Generic (PLEG): container finished" podID="7291d7dd-9676-440a-9b38-99eb02326cd1" containerID="f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d" exitCode=0 Oct 30 20:25:34.715881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.715779 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" Oct 30 20:25:34.715881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.715815 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" event={"ID":"7291d7dd-9676-440a-9b38-99eb02326cd1","Type":"ContainerDied","Data":"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d"} Oct 30 20:25:34.715881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.715865 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6" event={"ID":"7291d7dd-9676-440a-9b38-99eb02326cd1","Type":"ContainerDied","Data":"405d0f4571ca255372c5731260aee0b751ff5afc3d66a758f45d89fc74218cd0"} Oct 30 20:25:34.727550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.727524 4959 scope.go:122] "RemoveContainer" containerID="ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23" Oct 30 20:25:34.727898 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:34.727876 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23\": container with ID starting with ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23 not found: ID does not exist" containerID="ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23" Oct 30 20:25:34.727996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.727903 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23"} err="failed to get container status \"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23\": rpc error: code = NotFound desc = could not find container \"ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23\": container with ID starting with ccc3846bdc8cb32da70bff014d379ee9ae0bbbe79de5c5abee0d1ff712403f23 not found: ID does not exist" Oct 30 20:25:34.727996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.727920 4959 scope.go:122] "RemoveContainer" containerID="f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d" Oct 30 20:25:34.737587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.737570 4959 scope.go:122] "RemoveContainer" containerID="f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d" Oct 30 20:25:34.737848 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:34.737826 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d\": container with ID starting with f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d not found: ID does not exist" containerID="f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d" Oct 30 20:25:34.737908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.737860 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d"} err="failed to get container status \"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d\": rpc error: code = NotFound desc = could not find container \"f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d\": container with ID starting with f3be4d052d89b381c450ed81571f3429b4ed72334305024306eec3292177a60d not found: ID does not exist" Oct 30 20:25:34.773705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.773678 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:25:34.784234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.784215 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f35cc1-7d78c68f88-cq4s6"] Oct 30 20:25:34.806290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.806266 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:25:34.810919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.810900 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:34.818808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:34.818786 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f35cc1-74c6b56bd7-blp27"] Oct 30 20:25:35.449180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:35.449139 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:36.000106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:36.000076 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7291d7dd-9676-440a-9b38-99eb02326cd1" path="/var/lib/kubelet/pods/7291d7dd-9676-440a-9b38-99eb02326cd1/volumes" Oct 30 20:25:36.000474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:36.000460 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc92b9a6-e284-450a-becd-e88ba64484dc" path="/var/lib/kubelet/pods/fc92b9a6-e284-450a-becd-e88ba64484dc/volumes" Oct 30 20:25:36.802127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:36.802092 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" podUID="d99cfa0b-a981-49ec-a851-333206b31117" containerName="main" containerID="cri-o://14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511" gracePeriod=45 Oct 30 20:25:37.083232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.083206 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:25:37.143075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143042 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\" (UniqueName: \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143100 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143155 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143188 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143352 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\" (UniqueName: \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143359 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:37.143563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143425 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\") pod \"d99cfa0b-a981-49ec-a851-333206b31117\" (UID: \"d99cfa0b-a981-49ec-a851-333206b31117\") " Oct 30 20:25:37.143563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143406 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:37.143741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143689 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:37.143942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143909 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.143942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143929 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.143942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.143943 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.145395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.145373 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "kube-api-access-76xw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:37.145513 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.145433 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:37.145643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.145629 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs" pod "d99cfa0b-a981-49ec-a851-333206b31117" (UID: "d99cfa0b-a981-49ec-a851-333206b31117"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:37.245165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.245138 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99cfa0b-a981-49ec-a851-333206b31117-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.245165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.245165 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xw9\" (UniqueName: \"kubernetes.io/projected/d99cfa0b-a981-49ec-a851-333206b31117-kube-api-access-76xw9\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.245165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.245174 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99cfa0b-a981-49ec-a851-333206b31117-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:37.249819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.249792 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:37.740275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.740237 4959 generic.go:356] "Generic (PLEG): container finished" podID="d99cfa0b-a981-49ec-a851-333206b31117" containerID="14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511" exitCode=0 Oct 30 20:25:37.740427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.740292 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" Oct 30 20:25:37.740427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.740291 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" event={"ID":"d99cfa0b-a981-49ec-a851-333206b31117","Type":"ContainerDied","Data":"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511"} Oct 30 20:25:37.740427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.740341 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76" event={"ID":"d99cfa0b-a981-49ec-a851-333206b31117","Type":"ContainerDied","Data":"2b26dbb5b327f9d6797b49a62b681001eb28da462e60cadf9c72a793e298d163"} Oct 30 20:25:37.740427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.740375 4959 scope.go:122] "RemoveContainer" containerID="14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511" Oct 30 20:25:37.751360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.751339 4959 scope.go:122] "RemoveContainer" containerID="14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511" Oct 30 20:25:37.751704 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:37.751681 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511\": container with ID starting with 14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511 not found: ID does not exist" containerID="14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511" Oct 30 20:25:37.751773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.751712 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511"} err="failed to get container status \"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511\": rpc error: code = NotFound desc = could not find container \"14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511\": container with ID starting with 14c7cb055edbb891ab83ddd4a3668642a5c90ddfe0fad3e64f6e562ca919e511 not found: ID does not exist" Oct 30 20:25:37.790033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.790007 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:25:37.800744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.800723 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5314c2/auth-test-model-5314c2-kserve-996f5b5df-6xz76"] Oct 30 20:25:37.998164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:37.998122 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99cfa0b-a981-49ec-a851-333206b31117" path="/var/lib/kubelet/pods/d99cfa0b-a981-49ec-a851-333206b31117/volumes" Oct 30 20:25:39.596672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:39.596639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:40.558088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:40.558056 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 20:25:40.698753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:40.698719 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:40.816146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:40.811292 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:41.327742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:41.327716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:43.121814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:43.121782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:44.347463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:44.347433 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:25:44.385978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:44.385947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:44.964725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:44.964688 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerName="main" containerID="cri-o://a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f" gracePeriod=45 Oct 30 20:25:45.048769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.048741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:45.067880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.067842 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.197:8000/ready\": dial tcp 10.132.0.197:8000: connect: connection refused" Oct 30 20:25:45.282306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.282279 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:25:45.374756 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.374723 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\" (UniqueName: \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.374802 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.374908 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.374948 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.374992 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375044 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\") pod \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\" (UID: \"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e\") " Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375103 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:45.375261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375152 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:45.375872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375366 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:45.375872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375725 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.375872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375745 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.375872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.375758 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.377855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.377829 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:45.377968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.377858 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "kube-api-access-m8d8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:45.378778 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.378760 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm" pod "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" (UID: "a46fd4b2-f220-4ef8-8ec8-7f0359247c4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:45.476449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.476422 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.476449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.476447 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.476635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.476459 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8d8x\" (UniqueName: \"kubernetes.io/projected/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e-kube-api-access-m8d8x\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.558541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.558456 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" podUID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" containerName="payload-processing" containerID="cri-o://09375439bb2371942b295f1181747ab437939719afe2fc60f9f8512e67d30fbe" gracePeriod=25 Oct 30 20:25:45.794004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.793969 4959 generic.go:356] "Generic (PLEG): container finished" podID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerID="a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f" exitCode=0 Oct 30 20:25:45.794182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.794016 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" Oct 30 20:25:45.794182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.794053 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" event={"ID":"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e","Type":"ContainerDied","Data":"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f"} Oct 30 20:25:45.794182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.794094 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q" event={"ID":"a46fd4b2-f220-4ef8-8ec8-7f0359247c4e","Type":"ContainerDied","Data":"4f21da8e5673a3868ecee8119ff74cf1074ed40d7497f253d1d5bacabe5f34b8"} Oct 30 20:25:45.794182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.794110 4959 scope.go:122] "RemoveContainer" containerID="a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f" Oct 30 20:25:45.796547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.796481 4959 generic.go:356] "Generic (PLEG): container finished" podID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" containerID="09375439bb2371942b295f1181747ab437939719afe2fc60f9f8512e67d30fbe" exitCode=0 Oct 30 20:25:45.796699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.796608 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" event={"ID":"21ad2e0e-8576-4f37-931f-ce70b65e4e35","Type":"ContainerDied","Data":"09375439bb2371942b295f1181747ab437939719afe2fc60f9f8512e67d30fbe"} Oct 30 20:25:45.805544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.805524 4959 scope.go:122] "RemoveContainer" containerID="a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f" Oct 30 20:25:45.805801 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:45.805783 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f\": container with ID starting with a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f not found: ID does not exist" containerID="a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f" Oct 30 20:25:45.805860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.805809 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f"} err="failed to get container status \"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f\": rpc error: code = NotFound desc = could not find container \"a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f\": container with ID starting with a0c0f4a32a540f5ecb881750b157fd5977c49a692f40a9fe5867ba2994a6374f not found: ID does not exist" Oct 30 20:25:45.823288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.823272 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 20:25:45.847517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.847480 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:25:45.858415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.858381 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5c875d/auth-test-model-5c875d-kserve-5f998898c7-9vb8q"] Oct 30 20:25:45.880589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.880567 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\") pod \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " Oct 30 20:25:45.880703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.880673 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\" (UniqueName: \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\") pod \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\" (UID: \"21ad2e0e-8576-4f37-931f-ce70b65e4e35\") " Oct 30 20:25:45.880877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.880853 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume" pod "21ad2e0e-8576-4f37-931f-ce70b65e4e35" (UID: "21ad2e0e-8576-4f37-931f-ce70b65e4e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:25:45.881195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.881174 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ad2e0e-8576-4f37-931f-ce70b65e4e35-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.882886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.882862 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8" pod "21ad2e0e-8576-4f37-931f-ce70b65e4e35" (UID: "21ad2e0e-8576-4f37-931f-ce70b65e4e35"). InnerVolumeSpecName "kube-api-access-g7fj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:45.982586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.982551 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fj8\" (UniqueName: \"kubernetes.io/projected/21ad2e0e-8576-4f37-931f-ce70b65e4e35-kube-api-access-g7fj8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:45.998436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:45.998408 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" path="/var/lib/kubelet/pods/a46fd4b2-f220-4ef8-8ec8-7f0359247c4e/volumes" Oct 30 20:25:46.065257 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:46.065195 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:25:46.805214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.805180 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" event={"ID":"21ad2e0e-8576-4f37-931f-ce70b65e4e35","Type":"ContainerDied","Data":"e3a84cbde477e791be8270e1701dce11e4bae92baff2f2fbbc9d5dd2e1116980"} Oct 30 20:25:46.805716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.805239 4959 scope.go:122] "RemoveContainer" containerID="09375439bb2371942b295f1181747ab437939719afe2fc60f9f8512e67d30fbe" Oct 30 20:25:46.805716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.805195 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w" Oct 30 20:25:46.816984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.816941 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:46.854081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.854052 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 20:25:46.873544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:46.871199 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6c4dd1-59f965f976-q6w9w"] Oct 30 20:25:47.614336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:47.614303 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:47.996987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:47.996957 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" path="/var/lib/kubelet/pods/21ad2e0e-8576-4f37-931f-ce70b65e4e35/volumes" Oct 30 20:25:48.808930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:48.808900 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:50.730939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:50.730903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:50.859102 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:50.859067 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:52.396208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:52.396173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:53.645835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:53.645787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:53.959435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:53.959359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:56.457319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457285 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:25:56.457955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457936 4959 cpu_manager.go:414] "containerMap: removing container" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" Oct 30 20:25:56.457955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457956 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457976 4959 cpu_manager.go:414] "containerMap: removing container" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457983 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.457995 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fc92b9a6-e284-450a-becd-e88ba64484dc" containerName="payload-pre-processing" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458003 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc92b9a6-e284-450a-becd-e88ba64484dc" containerName="payload-pre-processing" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458023 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" containerName="maas-api" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458027 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" containerName="maas-api" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458043 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458048 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1d9453-4769-4ab6-8b80-b767c3ba2bd8" containerName="istio-proxy" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458054 4959 cpu_manager.go:414] "containerMap: removing container" podUID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" containerName="payload-processing" Oct 30 20:25:56.458057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458061 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458066 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458071 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458082 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7291d7dd-9676-440a-9b38-99eb02326cd1" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458086 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7291d7dd-9676-440a-9b38-99eb02326cd1" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458097 4959 cpu_manager.go:414] "containerMap: removing container" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458101 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458109 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d99cfa0b-a981-49ec-a851-333206b31117" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458113 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99cfa0b-a981-49ec-a851-333206b31117" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458201 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d243ba-2779-4a6c-abd0-498701701489" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458208 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99cfa0b-a981-49ec-a851-333206b31117" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458217 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e5ea21-98c1-4a84-9532-da59c98c4fe2" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458225 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="333d21e4-88f9-4bab-997f-2c82518767a2" containerName="payload-pre-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458231 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc92b9a6-e284-450a-becd-e88ba64484dc" containerName="payload-pre-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458240 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee95dd6-e2a1-437d-a529-d1a50e4096ad" containerName="maas-api" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458248 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ad2e0e-8576-4f37-931f-ce70b65e4e35" containerName="payload-processing" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458252 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46fd4b2-f220-4ef8-8ec8-7f0359247c4e" containerName="main" Oct 30 20:25:56.458387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.458257 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7291d7dd-9676-440a-9b38-99eb02326cd1" containerName="payload-processing" Oct 30 20:25:56.463456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.463436 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.478198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.478178 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5c875d-openshift-default-dockercfg-crmft\"" Oct 30 20:25:56.492893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.492873 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:25:56.640972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.640939 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.640988 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslfl\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641109 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641150 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641176 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641283 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641301 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641321 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.641380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.641345 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.675158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.675135 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:56.742407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.742323 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.742407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.742361 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.742407 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.742387 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.742998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.742965 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743072 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743115 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743173 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslfl\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743288 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743382 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743389 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.743532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.743447 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.744061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.744035 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.744182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.744125 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.744182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.744127 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.746449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.746418 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.746740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.746717 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.763380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.763354 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.763482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.763446 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslfl\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\") pod \"e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:56.777309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:56.777290 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:57.105747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.105671 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:25:57.110282 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:25:57.110253 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd591a945_0928_4519_ad6a_52c378a44d51.slice/crio-dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c WatchSource:0}: Error finding container dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c: Status 404 returned error can't find the container with id dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c Oct 30 20:25:57.113097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.113063 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:25:57.113184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.113137 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:25:57.113184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.113167 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:25:57.904067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.904019 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" event={"ID":"d591a945-0928-4519-ad6a-52c378a44d51","Type":"ContainerStarted","Data":"cd61ddedc7f8ca9000c06ce54d108d0ca38de5738f34d869a434294d0efd1fce"} Oct 30 20:25:57.904621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.904099 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" event={"ID":"d591a945-0928-4519-ad6a-52c378a44d51","Type":"ContainerStarted","Data":"dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c"} Oct 30 20:25:57.993672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:57.993619 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podStartSLOduration=1.9936067240000002 podStartE2EDuration="1.993606724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:57.992536298 +0000 UTC m=+3082.121770236" watchObservedRunningTime="2026-10-30 20:25:57.993606724 +0000 UTC m=+3082.122840624" Oct 30 20:25:58.777936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:58.777906 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:25:58.779647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:58.779619 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:25:58.779781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:58.779676 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 20:25:59.015505 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.015453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:59.156085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.156051 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:59.374067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.374030 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerName="main" containerID="cri-o://02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe" gracePeriod=45 Oct 30 20:25:59.512375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.512341 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerName="main" probeResult="failure" output="Get \"https://10.132.0.189:8000/ready\": dial tcp 10.132.0.189:8000: connect: connection refused" Oct 30 20:25:59.693852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.693828 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:25:59.778197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.778120 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:25:59.778360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.778204 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 20:25:59.793664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.793637 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.793813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.793728 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.793813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.793794 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.793933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.793850 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.794060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794034 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.794133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794087 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\" (UniqueName: \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\") pod \"efd6afec-8257-468f-a4e9-6bbcd2061063\" (UID: \"efd6afec-8257-468f-a4e9-6bbcd2061063\") " Oct 30 20:25:59.794212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794185 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:59.794311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794289 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:59.794369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794301 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:59.794618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794592 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.794618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794617 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.794821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.794632 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.796708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.796686 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:25:59.796806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.796703 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "kube-api-access-bnxwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:25:59.796806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.796791 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs" pod "efd6afec-8257-468f-a4e9-6bbcd2061063" (UID: "efd6afec-8257-468f-a4e9-6bbcd2061063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:25:59.895364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.895328 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxwz\" (UniqueName: \"kubernetes.io/projected/efd6afec-8257-468f-a4e9-6bbcd2061063-kube-api-access-bnxwz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.895364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.895360 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd6afec-8257-468f-a4e9-6bbcd2061063-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.895364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.895371 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd6afec-8257-468f-a4e9-6bbcd2061063-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:25:59.918507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.918463 4959 generic.go:356] "Generic (PLEG): container finished" podID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerID="02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe" exitCode=0 Oct 30 20:25:59.918697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.918525 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" Oct 30 20:25:59.918697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.918579 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" event={"ID":"efd6afec-8257-468f-a4e9-6bbcd2061063","Type":"ContainerDied","Data":"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe"} Oct 30 20:25:59.918697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.918625 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g" event={"ID":"efd6afec-8257-468f-a4e9-6bbcd2061063","Type":"ContainerDied","Data":"65a5dbf5898517b20b4a3e4fe4c67e88a90bf68b1a7b3743b4faedb76330583f"} Oct 30 20:25:59.918697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.918650 4959 scope.go:122] "RemoveContainer" containerID="02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe" Oct 30 20:25:59.930508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.930481 4959 scope.go:122] "RemoveContainer" containerID="02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe" Oct 30 20:25:59.930808 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:25:59.930791 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe\": container with ID starting with 02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe not found: ID does not exist" containerID="02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe" Oct 30 20:25:59.930876 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.930816 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe"} err="failed to get container status \"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe\": rpc error: code = NotFound desc = could not find container \"02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe\": container with ID starting with 02664ff7baa6b8742b975ac48738b8208420b6b97ff3f03550b2c1477e08fbbe not found: ID does not exist" Oct 30 20:25:59.976058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.976023 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:25:59.979340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:25:59.979321 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-77ac5a/auth-test-model-77ac5a-kserve-576fd5fc69-wbp8g"] Oct 30 20:26:00.000112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:00.000084 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" path="/var/lib/kubelet/pods/efd6afec-8257-468f-a4e9-6bbcd2061063/volumes" Oct 30 20:26:00.778167 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:00.778131 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:26:00.778606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:00.778184 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 20:26:00.906976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:00.906928 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:01.778439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:01.778403 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:26:01.778889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:01.778463 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 20:26:02.778509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:02.778459 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:26:02.778939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:02.778582 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 20:26:03.157699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:03.157665 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:03.783064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:03.783034 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:26:03.783535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:03.783522 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:26:03.784220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:03.784201 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:26:06.129729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:06.129701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:07.359820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:07.359789 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:26:07.484475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:07.484433 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:07.597733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:07.597699 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:07.734944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:07.734912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:08.700091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:08.700053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:10.670160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:10.670130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:10.962325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:10.962169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:11.075669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:11.075623 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:12.114001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.113964 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:12.278878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.278843 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:12.360385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.360343 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" podUID="93386db5-1aeb-4d49-9efc-92113dc725ae" containerName="payload-pre-processing" containerID="cri-o://e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c" gracePeriod=25 Oct 30 20:26:12.545522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.539475 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:26:12.545522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.540523 4959 cpu_manager.go:414] "containerMap: removing container" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerName="main" Oct 30 20:26:12.545522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.540542 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerName="main" Oct 30 20:26:12.545522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.540699 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd6afec-8257-468f-a4e9-6bbcd2061063" containerName="main" Oct 30 20:26:12.551531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.548794 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.569359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.568751 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9d2720-openshift-default-dockercfg-nhcwg\"" Oct 30 20:26:12.580585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.580526 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:26:12.646618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646589 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646641 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5vz\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646730 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646868 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646788 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646882 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646916 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646950 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.646980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.646968 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.647051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.647006 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.706748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.706725 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:26:12.748250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748206 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748291 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748373 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5vz\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748454 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748443 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748567 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748753 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748805 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748851 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.748914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.748878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.750801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.749905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.750801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.750154 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.750990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.750886 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.751094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.751069 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.752832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.752796 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.755108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.755068 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.755228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.755177 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.770560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.770519 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5vz\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.770982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.770956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\") pod \"e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.849317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.849292 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\" (UniqueName: \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\") pod \"93386db5-1aeb-4d49-9efc-92113dc725ae\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " Oct 30 20:26:12.849457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.849443 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\") pod \"93386db5-1aeb-4d49-9efc-92113dc725ae\" (UID: \"93386db5-1aeb-4d49-9efc-92113dc725ae\") " Oct 30 20:26:12.849798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.849778 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume" pod "93386db5-1aeb-4d49-9efc-92113dc725ae" (UID: "93386db5-1aeb-4d49-9efc-92113dc725ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:12.850057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.849903 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93386db5-1aeb-4d49-9efc-92113dc725ae-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:12.851352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.851331 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp" pod "93386db5-1aeb-4d49-9efc-92113dc725ae" (UID: "93386db5-1aeb-4d49-9efc-92113dc725ae"). InnerVolumeSpecName "kube-api-access-kwckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:12.873525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.873504 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:12.951209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:12.951171 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwckp\" (UniqueName: \"kubernetes.io/projected/93386db5-1aeb-4d49-9efc-92113dc725ae-kube-api-access-kwckp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:13.007002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.006963 4959 generic.go:356] "Generic (PLEG): container finished" podID="93386db5-1aeb-4d49-9efc-92113dc725ae" containerID="e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c" exitCode=0 Oct 30 20:26:13.007002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.007003 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" Oct 30 20:26:13.007258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.007048 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" event={"ID":"93386db5-1aeb-4d49-9efc-92113dc725ae","Type":"ContainerDied","Data":"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c"} Oct 30 20:26:13.007258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.007100 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl" event={"ID":"93386db5-1aeb-4d49-9efc-92113dc725ae","Type":"ContainerDied","Data":"10b2e6b07402bdd16dfdd0887d6a1b327d9a908a2284d6221ecb24d2ab38f12d"} Oct 30 20:26:13.007258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.007124 4959 scope.go:122] "RemoveContainer" containerID="e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c" Oct 30 20:26:13.019844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.018765 4959 scope.go:122] "RemoveContainer" containerID="e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c" Oct 30 20:26:13.019844 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:26:13.019080 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c\": container with ID starting with e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c not found: ID does not exist" containerID="e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c" Oct 30 20:26:13.019844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.019111 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c"} err="failed to get container status \"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c\": rpc error: code = NotFound desc = could not find container \"e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c\": container with ID starting with e2c2df05f5c3c3f9aadbebabd77634608c40fae9496f61a707c66c78ac7d119c not found: ID does not exist" Oct 30 20:26:13.056558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.056534 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:26:13.064573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.064551 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c19563-74c9b498b7-kwlrl"] Oct 30 20:26:13.193659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.193599 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:26:13.197676 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:26:13.197655 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9362571_8f8f_4bde_ae48_95ceec8c60fd.slice/crio-7c3ab10cc7e1806a17c4b33efef87a1ab7fc15db42201c167723caf8f5edbdd2 WatchSource:0}: Error finding container 7c3ab10cc7e1806a17c4b33efef87a1ab7fc15db42201c167723caf8f5edbdd2: Status 404 returned error can't find the container with id 7c3ab10cc7e1806a17c4b33efef87a1ab7fc15db42201c167723caf8f5edbdd2 Oct 30 20:26:13.200220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.200190 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:13.200309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.200253 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:13.200309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.200282 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:13.400658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.400629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:13.960120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.960015 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:13.998733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:13.998693 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93386db5-1aeb-4d49-9efc-92113dc725ae" path="/var/lib/kubelet/pods/93386db5-1aeb-4d49-9efc-92113dc725ae/volumes" Oct 30 20:26:14.014737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.014699 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" event={"ID":"e9362571-8f8f-4bde-ae48-95ceec8c60fd","Type":"ContainerStarted","Data":"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579"} Oct 30 20:26:14.014911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.014763 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" event={"ID":"e9362571-8f8f-4bde-ae48-95ceec8c60fd","Type":"ContainerStarted","Data":"7c3ab10cc7e1806a17c4b33efef87a1ab7fc15db42201c167723caf8f5edbdd2"} Oct 30 20:26:14.073654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.073596 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podStartSLOduration=2.073580228 podStartE2EDuration="2.073580228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:14.072124116 +0000 UTC m=+3098.201358038" watchObservedRunningTime="2026-10-30 20:26:14.073580228 +0000 UTC m=+3098.202814148" Oct 30 20:26:14.873689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.873655 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:14.875618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.875592 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:26:14.875769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:14.875639 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 20:26:15.203727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:15.203682 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:15.318753 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:15.318720 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:15.873882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:15.873853 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:26:15.874305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:15.873901 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 20:26:16.085659 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:26:16.085625 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:26:16.874444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:16.874350 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:26:16.874444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:16.874425 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 20:26:17.041015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:17.040957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:17.873956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:17.873925 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:26:17.874170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:17.873983 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 20:26:18.829072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:18.829041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:18.874910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:18.874875 4959 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:26:18.875232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:18.875193 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 20:26:19.878430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:19.878401 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:19.878979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:19.878957 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:19.879681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:19.879666 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:26:20.572620 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:20.572577 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:20.683560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:20.683521 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:22.520418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:22.520359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:23.734039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.734004 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:26:23.734627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.734611 4959 cpu_manager.go:414] "containerMap: removing container" podUID="93386db5-1aeb-4d49-9efc-92113dc725ae" containerName="payload-pre-processing" Oct 30 20:26:23.734627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.734627 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93386db5-1aeb-4d49-9efc-92113dc725ae" containerName="payload-pre-processing" Oct 30 20:26:23.734762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.734753 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="93386db5-1aeb-4d49-9efc-92113dc725ae" containerName="payload-pre-processing" Oct 30 20:26:23.739570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.739555 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.762042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.762021 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:26:23.770006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.769987 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-188d45-openshift-default-dockercfg-tqmpj\"" Oct 30 20:26:23.902834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.902802 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.902842 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.902962 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903091 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkjp\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903127 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903145 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903179 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903207 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:23.903301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:23.903257 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004355 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004432 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkjp\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004457 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004475 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004552 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004577 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004603 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004680 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004720 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004772 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.004891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004832 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.005163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004893 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.005163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.004925 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.005590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.005485 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.007119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.007097 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.007232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.007209 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.023904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.023879 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.024537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.024485 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkjp\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\") pod \"e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.053971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.053951 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:24.265399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.265329 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:24.408934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.408911 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:26:24.412100 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:26:24.412070 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab9c04d_8cdc_4dd5_a148_1e68991f92cc.slice/crio-362d9429a4769c416c11fef94c05676ef66aac2c6a1038f83317dc99d86bc9b6 WatchSource:0}: Error finding container 362d9429a4769c416c11fef94c05676ef66aac2c6a1038f83317dc99d86bc9b6: Status 404 returned error can't find the container with id 362d9429a4769c416c11fef94c05676ef66aac2c6a1038f83317dc99d86bc9b6 Oct 30 20:26:24.414774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.414740 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:24.414950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.414818 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:24.414950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:24.414863 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:26:25.088990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:25.088958 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" event={"ID":"eab9c04d-8cdc-4dd5-a148-1e68991f92cc","Type":"ContainerStarted","Data":"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620"} Oct 30 20:26:25.089375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:25.089012 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" event={"ID":"eab9c04d-8cdc-4dd5-a148-1e68991f92cc","Type":"ContainerStarted","Data":"362d9429a4769c416c11fef94c05676ef66aac2c6a1038f83317dc99d86bc9b6"} Oct 30 20:26:25.144081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:25.143999 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podStartSLOduration=2.143981718 podStartE2EDuration="2.143981718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:25.14130233 +0000 UTC m=+3109.270536252" watchObservedRunningTime="2026-10-30 20:26:25.143981718 +0000 UTC m=+3109.273215640" Oct 30 20:26:26.075741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:26.075706 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:26.076375 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:26.076347 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:26.076564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:26.076401 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:26.120571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:26.120542 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:27.064520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:27.058279 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:27.064520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:27.058333 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:27.072907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:27.072878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:27.182552 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:27.182474 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:27.930847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:27.930744 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:28.072656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:28.072610 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:28.072856 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:28.072668 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:29.070522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:29.066816 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:29.070522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:29.066887 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:29.808675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:29.808641 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:30.072945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:30.072860 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:30.073329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:30.072935 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:31.054272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:31.054231 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 20:26:31.054482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:31.054299 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 20:26:31.966864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:31.966828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:32.058932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:32.058905 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:32.059286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:32.059267 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:32.059917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:32.059898 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:26:33.314163 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:33.314131 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:33.996957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:33.996887 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:35.995924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:35.995889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:38.495088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:38.495052 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:39.937142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:39.937108 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 20:26:39.938028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:39.937975 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" podUID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" containerName="maas-api" containerID="cri-o://13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239" gracePeriod=30 Oct 30 20:26:40.000814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.000754 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:40.263900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.263873 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 20:26:40.280942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.280902 4959 generic.go:356] "Generic (PLEG): container finished" podID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" containerID="13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239" exitCode=0 Oct 30 20:26:40.281100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.280947 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" Oct 30 20:26:40.281100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.280988 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" event={"ID":"c0157790-cbb5-4d18-8946-a6af70e6d6a2","Type":"ContainerDied","Data":"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239"} Oct 30 20:26:40.281100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.281054 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz" event={"ID":"c0157790-cbb5-4d18-8946-a6af70e6d6a2","Type":"ContainerDied","Data":"12c4b0bd719631e8d6d48f2fe343f672b0c847f4f2a9da9cb3c503c954341a36"} Oct 30 20:26:40.281100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.281083 4959 scope.go:122] "RemoveContainer" containerID="13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239" Oct 30 20:26:40.293721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.293705 4959 scope.go:122] "RemoveContainer" containerID="13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239" Oct 30 20:26:40.294076 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:26:40.294050 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239\": container with ID starting with 13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239 not found: ID does not exist" containerID="13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239" Oct 30 20:26:40.294172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.294095 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239"} err="failed to get container status \"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239\": rpc error: code = NotFound desc = could not find container \"13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239\": container with ID starting with 13e953413eee1f2d21f14d9f6dcd2e01921558b5feb9a1b9c2ee4f06b94e0239 not found: ID does not exist" Oct 30 20:26:40.372069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372032 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\") pod \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " Oct 30 20:26:40.372273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372108 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\" (UniqueName: \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\") pod \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " Oct 30 20:26:40.372342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372301 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\") pod \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " Oct 30 20:26:40.372402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372379 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\") pod \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\" (UID: \"c0157790-cbb5-4d18-8946-a6af70e6d6a2\") " Oct 30 20:26:40.372687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372655 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca" pod "c0157790-cbb5-4d18-8946-a6af70e6d6a2" (UID: "c0157790-cbb5-4d18-8946-a6af70e6d6a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:40.373018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.372894 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0157790-cbb5-4d18-8946-a6af70e6d6a2-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:40.374453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.374427 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls" pod "c0157790-cbb5-4d18-8946-a6af70e6d6a2" (UID: "c0157790-cbb5-4d18-8946-a6af70e6d6a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:40.374572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.374461 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x" pod "c0157790-cbb5-4d18-8946-a6af70e6d6a2" (UID: "c0157790-cbb5-4d18-8946-a6af70e6d6a2"). InnerVolumeSpecName "kube-api-access-dmw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:40.374777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.374752 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs" pod "c0157790-cbb5-4d18-8946-a6af70e6d6a2" (UID: "c0157790-cbb5-4d18-8946-a6af70e6d6a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:40.475734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.475607 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:40.475734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.475656 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmw9x\" (UniqueName: \"kubernetes.io/projected/c0157790-cbb5-4d18-8946-a6af70e6d6a2-kube-api-access-dmw9x\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:40.475734 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.475672 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0157790-cbb5-4d18-8946-a6af70e6d6a2-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:40.628022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.627992 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 20:26:40.634649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.634622 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8a3426-68f7d57985-57cfz"] Oct 30 20:26:40.699066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.699027 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:40.811223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.811189 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:40.933523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:40.930681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:41.403646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:41.403604 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:41.998944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:41.998913 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" path="/var/lib/kubelet/pods/c0157790-cbb5-4d18-8946-a6af70e6d6a2/volumes" Oct 30 20:26:42.782737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:42.782686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:44.267373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:44.267334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:44.444286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:44.444250 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:45.158157 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.158118 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 20:26:45.158659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.158630 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" podUID="50891422-97f8-4a2a-a2e9-bae193d83192" containerName="maas-api" containerID="cri-o://ff37e44690340d8a1e8e9a598e82eb6dad87a03097f108e217b4e3def3bfb4fb" gracePeriod=30 Oct 30 20:26:45.225692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.225662 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:45.381743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.381710 4959 generic.go:356] "Generic (PLEG): container finished" podID="50891422-97f8-4a2a-a2e9-bae193d83192" containerID="ff37e44690340d8a1e8e9a598e82eb6dad87a03097f108e217b4e3def3bfb4fb" exitCode=0 Oct 30 20:26:45.382124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.381789 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" event={"ID":"50891422-97f8-4a2a-a2e9-bae193d83192","Type":"ContainerDied","Data":"ff37e44690340d8a1e8e9a598e82eb6dad87a03097f108e217b4e3def3bfb4fb"} Oct 30 20:26:45.513928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.513902 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 20:26:45.685990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.685951 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\") pod \"50891422-97f8-4a2a-a2e9-bae193d83192\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " Oct 30 20:26:45.686201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.686015 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\" (UniqueName: \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\") pod \"50891422-97f8-4a2a-a2e9-bae193d83192\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " Oct 30 20:26:45.686201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.686062 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\") pod \"50891422-97f8-4a2a-a2e9-bae193d83192\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " Oct 30 20:26:45.686201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.686113 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\") pod \"50891422-97f8-4a2a-a2e9-bae193d83192\" (UID: \"50891422-97f8-4a2a-a2e9-bae193d83192\") " Oct 30 20:26:45.686518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.686466 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca" pod "50891422-97f8-4a2a-a2e9-bae193d83192" (UID: "50891422-97f8-4a2a-a2e9-bae193d83192"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:45.686851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.686830 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50891422-97f8-4a2a-a2e9-bae193d83192-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:45.688692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.688669 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls" pod "50891422-97f8-4a2a-a2e9-bae193d83192" (UID: "50891422-97f8-4a2a-a2e9-bae193d83192"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:45.688990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.688970 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs" pod "50891422-97f8-4a2a-a2e9-bae193d83192" (UID: "50891422-97f8-4a2a-a2e9-bae193d83192"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:26:45.689043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.689004 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4" pod "50891422-97f8-4a2a-a2e9-bae193d83192" (UID: "50891422-97f8-4a2a-a2e9-bae193d83192"). InnerVolumeSpecName "kube-api-access-dpzk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:45.788238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.788150 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:45.788238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.788178 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50891422-97f8-4a2a-a2e9-bae193d83192-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:45.788238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.788189 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpzk4\" (UniqueName: \"kubernetes.io/projected/50891422-97f8-4a2a-a2e9-bae193d83192-kube-api-access-dpzk4\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:26:45.924164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:45.924132 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:46.059428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.059343 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:46.066308 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:26:46.066279 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:26:46.269104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.269074 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:46.398935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.398907 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" Oct 30 20:26:46.399418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.398909 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k" event={"ID":"50891422-97f8-4a2a-a2e9-bae193d83192","Type":"ContainerDied","Data":"f59cecbd104457807dcf0fc015ad3e7c2e28e9b904fc5e8338aebd21d9774dc8"} Oct 30 20:26:46.399418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.399020 4959 scope.go:122] "RemoveContainer" containerID="ff37e44690340d8a1e8e9a598e82eb6dad87a03097f108e217b4e3def3bfb4fb" Oct 30 20:26:46.505276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.505243 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 20:26:46.508664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.508633 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:46.516703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.516681 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6c4dd1-6d759db9b4-hv85k"] Oct 30 20:26:46.620005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:46.619973 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:47.997384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:47.997353 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50891422-97f8-4a2a-a2e9-bae193d83192" path="/var/lib/kubelet/pods/50891422-97f8-4a2a-a2e9-bae193d83192/volumes" Oct 30 20:26:48.082255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:48.082225 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:48.192448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:48.192418 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:48.454090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:48.454059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:48.932757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:48.932725 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:50.711547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:50.711516 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:51.955154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:51.955122 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:52.803205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:52.803172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:53.960533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:53.960504 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:55.767669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:55.767618 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:55.990845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:55.990804 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:56.155714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:56.155678 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:56.389802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:56.389765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:57.266221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:57.266183 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:58.023509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:58.023469 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:59.912051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:26:59.912021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.208752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:00.208667 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.694383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:00.694334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:01.117646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:01.117548 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:01.870379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:01.870347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:03.713092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:03.713055 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:04.690894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:04.690857 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:04.800897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:04.800865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:05.702744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:05.702709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:06.790026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.789993 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:06.899852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.899822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:06.968933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.968904 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:27:06.969615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969595 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" containerName="maas-api" Oct 30 20:27:06.969744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969616 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" containerName="maas-api" Oct 30 20:27:06.969744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969682 4959 cpu_manager.go:414] "containerMap: removing container" podUID="50891422-97f8-4a2a-a2e9-bae193d83192" containerName="maas-api" Oct 30 20:27:06.969744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969691 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50891422-97f8-4a2a-a2e9-bae193d83192" containerName="maas-api" Oct 30 20:27:06.969915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969862 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0157790-cbb5-4d18-8946-a6af70e6d6a2" containerName="maas-api" Oct 30 20:27:06.969915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.969887 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="50891422-97f8-4a2a-a2e9-bae193d83192" containerName="maas-api" Oct 30 20:27:06.973305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.973284 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:06.989717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:06.989697 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-843ea8\"" Oct 30 20:27:07.000952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.000915 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-843ea8-dockercfg-hvjhw\"" Oct 30 20:27:07.004153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.004122 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.005532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.004570 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqrg\" (UniqueName: \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.005532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.005037 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:27:07.119656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.116194 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.119656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.116454 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqrg\" (UniqueName: \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.120855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.120821 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.171121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.171084 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqrg\" (UniqueName: \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\") pod \"payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.175690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.175661 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:27:07.181360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.181212 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:27:07.181360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.181306 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.217601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.217572 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5nj\" (UniqueName: \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.217809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.217767 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.289334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.289295 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:07.319167 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.319136 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5nj\" (UniqueName: \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.319331 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.319288 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.320066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.320044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.342239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.342208 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5nj\" (UniqueName: \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\") pod \"payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.497579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.497545 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:07.605559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.605526 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:27:07.612271 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:27:07.612240 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bde44b_797a_4bb4_855e_c23e21257189.slice/crio-c895c23fb505ee801ff7fff4074843065458cf24c4b8b66e83dc8f85a90d6eb1 WatchSource:0}: Error finding container c895c23fb505ee801ff7fff4074843065458cf24c4b8b66e83dc8f85a90d6eb1: Status 404 returned error can't find the container with id c895c23fb505ee801ff7fff4074843065458cf24c4b8b66e83dc8f85a90d6eb1 Oct 30 20:27:07.805621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.805470 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:27:07.808725 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:27:07.808694 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcd9b57_b4e9_4eaa_8819_4cdb8e27a5cb.slice/crio-db97ab7c56e7c88630ec9775089bc950b9721123610c3134175cfe31cc7295b6 WatchSource:0}: Error finding container db97ab7c56e7c88630ec9775089bc950b9721123610c3134175cfe31cc7295b6: Status 404 returned error can't find the container with id db97ab7c56e7c88630ec9775089bc950b9721123610c3134175cfe31cc7295b6 Oct 30 20:27:07.869919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:07.869886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:08.539673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.539632 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" event={"ID":"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb","Type":"ContainerStarted","Data":"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f"} Oct 30 20:27:08.539889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.539690 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" event={"ID":"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb","Type":"ContainerStarted","Data":"db97ab7c56e7c88630ec9775089bc950b9721123610c3134175cfe31cc7295b6"} Oct 30 20:27:08.539889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.539847 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:08.541484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.541456 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" event={"ID":"f1bde44b-797a-4bb4-855e-c23e21257189","Type":"ContainerStarted","Data":"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec"} Oct 30 20:27:08.541631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.541527 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:08.541631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.541547 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" event={"ID":"f1bde44b-797a-4bb4-855e-c23e21257189","Type":"ContainerStarted","Data":"c895c23fb505ee801ff7fff4074843065458cf24c4b8b66e83dc8f85a90d6eb1"} Oct 30 20:27:08.577776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.577723 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" podStartSLOduration=1.577712786 podStartE2EDuration="1.577712786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:08.577464765 +0000 UTC m=+3152.706698686" watchObservedRunningTime="2026-10-30 20:27:08.577712786 +0000 UTC m=+3152.706946706" Oct 30 20:27:08.616429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:08.616373 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" podStartSLOduration=2.616356531 podStartE2EDuration="2.616356531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:08.613292528 +0000 UTC m=+3152.742526449" watchObservedRunningTime="2026-10-30 20:27:08.616356531 +0000 UTC m=+3152.745590449" Oct 30 20:27:09.606236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:09.606188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:10.993911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:10.993881 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.187156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:11.187114 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.300125 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:11.300040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.681094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:11.681056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.790242 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:11.790201 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.895082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:11.895032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:12.320821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:12.320786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:13.691368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:13.691335 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:15.386676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:15.386648 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:16.066719 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:27:16.066684 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:27:17.017989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:17.017954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:17.357982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:17.357901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.421957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.421922 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:27:18.423223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.422683 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" podUID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" containerName="maas-api" containerID="cri-o://874fd9db07e778a9cde3eafbf6868ad568cd022161d315cbf78ba16db28acbcb" gracePeriod=30 Oct 30 20:27:18.496196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.496167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.617021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.616989 4959 generic.go:356] "Generic (PLEG): container finished" podID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" containerID="874fd9db07e778a9cde3eafbf6868ad568cd022161d315cbf78ba16db28acbcb" exitCode=0 Oct 30 20:27:18.617254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.617070 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" event={"ID":"35c9f14a-fae3-4dc8-bfba-edb439bd763a","Type":"ContainerDied","Data":"874fd9db07e778a9cde3eafbf6868ad568cd022161d315cbf78ba16db28acbcb"} Oct 30 20:27:18.709252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.709225 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:27:18.752675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.752636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.798797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.798756 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\" (UniqueName: \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\") pod \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " Oct 30 20:27:18.799060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.798844 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\") pod \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " Oct 30 20:27:18.799060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.798897 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") pod \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " Oct 30 20:27:18.799274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.799077 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\") pod \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\" (UID: \"35c9f14a-fae3-4dc8-bfba-edb439bd763a\") " Oct 30 20:27:18.799274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.799224 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca" pod "35c9f14a-fae3-4dc8-bfba-edb439bd763a" (UID: "35c9f14a-fae3-4dc8-bfba-edb439bd763a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:27:18.799672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.799638 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c9f14a-fae3-4dc8-bfba-edb439bd763a-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:27:18.801583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.801540 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7" pod "35c9f14a-fae3-4dc8-bfba-edb439bd763a" (UID: "35c9f14a-fae3-4dc8-bfba-edb439bd763a"). InnerVolumeSpecName "kube-api-access-gmgd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:27:18.801933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.801901 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs" pod "35c9f14a-fae3-4dc8-bfba-edb439bd763a" (UID: "35c9f14a-fae3-4dc8-bfba-edb439bd763a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:18.802043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.801990 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls" pod "35c9f14a-fae3-4dc8-bfba-edb439bd763a" (UID: "35c9f14a-fae3-4dc8-bfba-edb439bd763a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:27:18.868842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.868801 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.901228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.901193 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgd7\" (UniqueName: \"kubernetes.io/projected/35c9f14a-fae3-4dc8-bfba-edb439bd763a-kube-api-access-gmgd7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:27:18.901228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.901226 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:27:18.901228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:18.901235 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c9f14a-fae3-4dc8-bfba-edb439bd763a-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:27:19.193444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.193408 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:19.325981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.325947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:19.439889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.439849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:19.551100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.551023 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:27:19.551100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.551100 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:27:19.625360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.625331 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" Oct 30 20:27:19.625606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.625365 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx" event={"ID":"35c9f14a-fae3-4dc8-bfba-edb439bd763a","Type":"ContainerDied","Data":"a3a9cb2e33fcda6289aa21180609982534a90bc26fff04279d9bf1755d76b813"} Oct 30 20:27:19.625606 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.625432 4959 scope.go:122] "RemoveContainer" containerID="874fd9db07e778a9cde3eafbf6868ad568cd022161d315cbf78ba16db28acbcb" Oct 30 20:27:19.707440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.707333 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:27:19.711995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.711964 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5fbda3-597ccbcfb5-l44dx"] Oct 30 20:27:19.844007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:19.843915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:20.000321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:20.000285 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" path="/var/lib/kubelet/pods/35c9f14a-fae3-4dc8-bfba-edb439bd763a/volumes" Oct 30 20:27:21.197789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:21.197757 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:23.054021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:23.053985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:25.326925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:25.326890 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:26.032659 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:26.032630 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:26.139685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:26.139637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:26.395597 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:26.395562 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:27.629276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:27.629234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:29.953869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:29.953831 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:31.752782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:31.752739 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:33.481415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:33.481367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:34.079220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:34.079158 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:35.810332 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:35.810285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:37.493045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:37.493010 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:39.253654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:39.253620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:41.157174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:41.157140 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:41.263117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:41.263080 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:41.369739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:41.369703 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:42.989929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:42.989883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:44.645363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:44.645318 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:46.065322 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:27:46.065284 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:27:46.307826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:46.307737 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:47.937463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:47.937432 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:49.609679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:49.609645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:51.332637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:51.332583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:52.994205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:52.994173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.752340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:53.752302 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:55.665419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:55.665385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:55.781564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:55.781534 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:56.390815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:56.390787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:58.128940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:58.128901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:59.858183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:27:59.858148 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.189898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:01.189865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.303238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:01.303211 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.491054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:01.490973 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:28:01.491257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:01.491231 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" containerID="cri-o://e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620" gracePeriod=30 Oct 30 20:28:01.537769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:01.537736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:02.061860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:02.061831 4959 patch_prober.go:29] interesting pod/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:28:02.062028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:02.061890 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:28:02.252183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:02.252141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:04.500832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:04.500794 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:06.208318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.208273 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:06.778085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.778055 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:28:06.809776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.809739 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.809938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.809793 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.809938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.809837 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.809938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.809884 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.809938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.809916 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.810168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810008 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.810168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810045 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:06.810168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810112 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.810168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810151 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.810408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810218 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:06.810408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810227 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\") pod \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\" (UID: \"eab9c04d-8cdc-4dd5-a148-1e68991f92cc\") " Oct 30 20:28:06.810408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810242 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:06.810870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810579 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:06.810870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.810720 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:06.811129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.811090 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.811129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.811122 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.811273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.811138 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.811273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.811155 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.811273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.811170 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.812321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.812298 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:06.812480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.812460 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:28:06.813065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.813046 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "kube-api-access-tkkjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:06.813148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.813080 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token" pod "eab9c04d-8cdc-4dd5-a148-1e68991f92cc" (UID: "eab9c04d-8cdc-4dd5-a148-1e68991f92cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:06.911990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.911951 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.912161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.912007 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkkjp\" (UniqueName: \"kubernetes.io/projected/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-kube-api-access-tkkjp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.912161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.912055 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.912161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.912069 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab9c04d-8cdc-4dd5-a148-1e68991f92cc-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:06.974190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.974161 4959 generic.go:356] "Generic (PLEG): container finished" podID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerID="e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620" exitCode=0 Oct 30 20:28:06.974383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.974226 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" Oct 30 20:28:06.974383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.974251 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" event={"ID":"eab9c04d-8cdc-4dd5-a148-1e68991f92cc","Type":"ContainerDied","Data":"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620"} Oct 30 20:28:06.974383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.974340 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5" event={"ID":"eab9c04d-8cdc-4dd5-a148-1e68991f92cc","Type":"ContainerDied","Data":"362d9429a4769c416c11fef94c05676ef66aac2c6a1038f83317dc99d86bc9b6"} Oct 30 20:28:06.974383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.974372 4959 scope.go:122] "RemoveContainer" containerID="e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620" Oct 30 20:28:06.986196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.986178 4959 scope.go:122] "RemoveContainer" containerID="e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620" Oct 30 20:28:06.986513 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:06.986470 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620\": container with ID starting with e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620 not found: ID does not exist" containerID="e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620" Oct 30 20:28:06.986678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:06.986652 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620"} err="failed to get container status \"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620\": rpc error: code = NotFound desc = could not find container \"e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620\": container with ID starting with e248849a921a7701f7e5093c245d8761f8eb992bf8994ab47504c2e28de26620 not found: ID does not exist" Oct 30 20:28:07.024071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.024022 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:28:07.028211 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.028192 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-188d45-openshift-default-574f7f4f75-584k5"] Oct 30 20:28:07.649633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.649591 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:28:07.650092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.649804 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" podUID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" containerName="maas-api" containerID="cri-o://97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977" gracePeriod=30 Oct 30 20:28:07.728226 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.728197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:07.936067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.936038 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:28:07.984288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.984232 4959 generic.go:356] "Generic (PLEG): container finished" podID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" containerID="97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977" exitCode=0 Oct 30 20:28:07.984471 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.984414 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" Oct 30 20:28:07.984471 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.984427 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" event={"ID":"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d","Type":"ContainerDied","Data":"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977"} Oct 30 20:28:07.984629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.984475 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp" event={"ID":"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d","Type":"ContainerDied","Data":"0329dd3b3fa12ed3b4396bd939e7c74c96bebf8fac3cf5cf4bd7496ee2c2c152"} Oct 30 20:28:07.984629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.984520 4959 scope.go:122] "RemoveContainer" containerID="97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977" Oct 30 20:28:08.004521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:07.997149 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.004521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.002940 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" path="/var/lib/kubelet/pods/eab9c04d-8cdc-4dd5-a148-1e68991f92cc/volumes" Oct 30 20:28:08.013039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.013019 4959 scope.go:122] "RemoveContainer" containerID="97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977" Oct 30 20:28:08.013391 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:08.013361 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977\": container with ID starting with 97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977 not found: ID does not exist" containerID="97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977" Oct 30 20:28:08.013543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.013400 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977"} err="failed to get container status \"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977\": rpc error: code = NotFound desc = could not find container \"97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977\": container with ID starting with 97062ba0cfa26a0410258da1d28c73b24ef7f3b27528bd8ed759292f4742f977 not found: ID does not exist" Oct 30 20:28:08.026010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.025968 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\" (UniqueName: \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\") pod \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " Oct 30 20:28:08.026148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.026081 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") pod \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " Oct 30 20:28:08.026225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.026171 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\") pod \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " Oct 30 20:28:08.026225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.026204 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\") pod \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\" (UID: \"345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d\") " Oct 30 20:28:08.026703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.026677 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca" pod "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" (UID: "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:08.028580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.028551 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs" pod "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" (UID: "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:08.028937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.028914 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n" pod "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" (UID: "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d"). InnerVolumeSpecName "kube-api-access-kdt2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:08.029016 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.028955 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls" pod "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" (UID: "345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:08.128267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.128223 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdt2n\" (UniqueName: \"kubernetes.io/projected/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-kube-api-access-kdt2n\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:08.128267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.128256 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:08.128267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.128271 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:08.128587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.128284 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:08.138333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.138274 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.358304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.358271 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:28:08.361768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.361739 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f35cc1-678fc8c544-pmmhp"] Oct 30 20:28:08.663085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.663047 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.776341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.776310 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.887402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:08.887375 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:09.276040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:09.276001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:10.009556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:10.009519 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" path="/var/lib/kubelet/pods/345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d/volumes" Oct 30 20:28:10.899172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:10.899130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:12.992595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:12.992543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:14.804911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:14.804877 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.306779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:15.306743 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.411583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:15.411541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.701986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:15.701955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:16.066353 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:16.066279 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:28:16.815185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:16.815154 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:18.624816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:18.624751 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:20.493867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:20.493836 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:21.233140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:21.233108 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:28:21.268118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:21.268090 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:28:21.365585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:21.365546 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:21.483310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:21.483208 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:21.599860 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:21.599810 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:22.300579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:22.300548 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:24.046902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:24.046871 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:25.846966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:25.846932 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:26.234452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.234402 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" podUID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" containerName="payload-processing" containerID="cri-o://101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f" gracePeriod=25 Oct 30 20:28:26.268505 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.268466 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" podUID="f1bde44b-797a-4bb4-855e-c23e21257189" containerName="payload-pre-processing" containerID="cri-o://4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec" gracePeriod=25 Oct 30 20:28:26.344998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.344972 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 20:28:26.375164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.375123 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 20:28:26.485364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.485263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:26.602399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.602299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:26.604679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.604316 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:28:26.629414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.629387 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:28:26.755537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.755429 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\") pod \"f1bde44b-797a-4bb4-855e-c23e21257189\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " Oct 30 20:28:26.755707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.755636 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\" (UniqueName: \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\") pod \"f1bde44b-797a-4bb4-855e-c23e21257189\" (UID: \"f1bde44b-797a-4bb4-855e-c23e21257189\") " Oct 30 20:28:26.755707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.755664 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\") pod \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " Oct 30 20:28:26.755818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.755772 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\" (UniqueName: \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\") pod \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\" (UID: \"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb\") " Oct 30 20:28:26.755899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.755873 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume" pod "f1bde44b-797a-4bb4-855e-c23e21257189" (UID: "f1bde44b-797a-4bb4-855e-c23e21257189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:26.756190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.756170 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume" pod "5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" (UID: "5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:26.756279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.756264 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1bde44b-797a-4bb4-855e-c23e21257189-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:26.756342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.756286 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:26.758229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.758205 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg" pod "f1bde44b-797a-4bb4-855e-c23e21257189" (UID: "f1bde44b-797a-4bb4-855e-c23e21257189"). InnerVolumeSpecName "kube-api-access-rkqrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:26.758339 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.758308 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj" pod "5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" (UID: "5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb"). InnerVolumeSpecName "kube-api-access-xc5nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:26.857383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.857362 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkqrg\" (UniqueName: \"kubernetes.io/projected/f1bde44b-797a-4bb4-855e-c23e21257189-kube-api-access-rkqrg\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:26.857383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:26.857383 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5nj\" (UniqueName: \"kubernetes.io/projected/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb-kube-api-access-xc5nj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:27.160559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.159786 4959 generic.go:356] "Generic (PLEG): container finished" podID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" containerID="101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f" exitCode=0 Oct 30 20:28:27.160559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.159913 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" event={"ID":"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb","Type":"ContainerDied","Data":"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f"} Oct 30 20:28:27.160559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.159962 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" event={"ID":"5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb","Type":"ContainerDied","Data":"db97ab7c56e7c88630ec9775089bc950b9721123610c3134175cfe31cc7295b6"} Oct 30 20:28:27.160559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.159987 4959 scope.go:122] "RemoveContainer" containerID="101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f" Oct 30 20:28:27.160559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.160135 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2" Oct 30 20:28:27.163841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.163811 4959 generic.go:356] "Generic (PLEG): container finished" podID="f1bde44b-797a-4bb4-855e-c23e21257189" containerID="4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec" exitCode=0 Oct 30 20:28:27.164017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.163885 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" event={"ID":"f1bde44b-797a-4bb4-855e-c23e21257189","Type":"ContainerDied","Data":"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec"} Oct 30 20:28:27.164017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.163927 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" event={"ID":"f1bde44b-797a-4bb4-855e-c23e21257189","Type":"ContainerDied","Data":"c895c23fb505ee801ff7fff4074843065458cf24c4b8b66e83dc8f85a90d6eb1"} Oct 30 20:28:27.164017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.163948 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2" Oct 30 20:28:27.175918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.175898 4959 scope.go:122] "RemoveContainer" containerID="101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f" Oct 30 20:28:27.176223 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:27.176206 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f\": container with ID starting with 101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f not found: ID does not exist" containerID="101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f" Oct 30 20:28:27.176301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.176229 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f"} err="failed to get container status \"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f\": rpc error: code = NotFound desc = could not find container \"101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f\": container with ID starting with 101befe09e3290ef2026c7188ec48a40a711b37792d67318279ae512e827528f not found: ID does not exist" Oct 30 20:28:27.176301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.176248 4959 scope.go:122] "RemoveContainer" containerID="4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec" Oct 30 20:28:27.185631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.185617 4959 scope.go:122] "RemoveContainer" containerID="4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec" Oct 30 20:28:27.185898 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:27.185879 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec\": container with ID starting with 4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec not found: ID does not exist" containerID="4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec" Oct 30 20:28:27.185966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.185904 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec"} err="failed to get container status \"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec\": rpc error: code = NotFound desc = could not find container \"4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec\": container with ID starting with 4b7dd23158a8058aa0632d14d19806e0366dbfa41c95ed476e376a28bb8537ec not found: ID does not exist" Oct 30 20:28:27.221225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.221203 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:28:27.232019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.231993 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-843ea8-7b848c58dd-swqt2"] Oct 30 20:28:27.257263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.256810 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:28:27.259976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.259941 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-843ea8-5fc9b59d8c-m5kv2"] Oct 30 20:28:27.626869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.626791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:27.997018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.996983 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" path="/var/lib/kubelet/pods/5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb/volumes" Oct 30 20:28:27.997541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:27.997379 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bde44b-797a-4bb4-855e-c23e21257189" path="/var/lib/kubelet/pods/f1bde44b-797a-4bb4-855e-c23e21257189/volumes" Oct 30 20:28:29.324717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:29.324672 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:30.925795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:30.925761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:31.345631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:31.345516 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" containerID="cri-o://e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4" gracePeriod=25 Oct 30 20:28:31.376050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:31.376011 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" containerID="cri-o://3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e" gracePeriod=25 Oct 30 20:28:32.599353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:32.599322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:33.565488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:33.565458 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:34.137814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:34.137775 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:34.309203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:34.309128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:36.051268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:36.051238 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.036341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.036269 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.220565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.220538 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.328779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.328709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.590849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.590765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.705053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.705008 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:38.813658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:38.813593 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:39.231320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:39.231291 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:40.246690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:40.246655 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:40.553305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:40.553184 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:41.277351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:41.277315 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:42.312696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:42.312652 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:43.997199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:43.997140 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:45.639142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:45.639111 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:45.763998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:45.763930 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:45.977700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:45.977595 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:28:45.979578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:45.977874 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" containerID="cri-o://c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020" gracePeriod=30 Oct 30 20:28:46.028812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:46.028779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:46.066418 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:46.066386 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:28:47.491090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:47.491058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:49.351655 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:49.351621 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:50.246103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:50.246061 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:50.526489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:50.526416 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:50.647248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:50.647213 4959 patch_prober.go:29] interesting pod/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:28:50.647547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:50.647517 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:28:50.685696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:50.685670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:51.265275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.265215 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:28:51.275506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275471 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275559 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275592 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275664 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275689 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275720 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.275786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275755 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.276014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275789 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.276014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275886 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\") pod \"618a7097-47ef-4d5b-8474-4353dface889\" (UID: \"618a7097-47ef-4d5b-8474-4353dface889\") " Oct 30 20:28:51.276014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275922 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:51.276014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275958 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:51.276014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.275974 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:51.276365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276046 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:51.276613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276570 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:51.276759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276691 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.276759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276716 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.276759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276734 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.276759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276748 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.277003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.276761 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/618a7097-47ef-4d5b-8474-4353dface889-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.277903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.277871 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:51.278368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.278343 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:51.278485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.278447 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:51.278724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.278562 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:28:51.278934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.278915 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq" pod "618a7097-47ef-4d5b-8474-4353dface889" (UID: "618a7097-47ef-4d5b-8474-4353dface889"). InnerVolumeSpecName "kube-api-access-rf8fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:51.316527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.316484 4959 generic.go:356] "Generic (PLEG): container finished" podID="618a7097-47ef-4d5b-8474-4353dface889" containerID="c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020" exitCode=0 Oct 30 20:28:51.316686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.316542 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" Oct 30 20:28:51.316686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.316549 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerDied","Data":"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020"} Oct 30 20:28:51.316686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.316660 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm" event={"ID":"618a7097-47ef-4d5b-8474-4353dface889","Type":"ContainerDied","Data":"eaf4853141ea95cfd0c429bf69f26265c2fb8d29ac5940ff9448e755ecd9cac0"} Oct 30 20:28:51.316686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.316681 4959 scope.go:122] "RemoveContainer" containerID="c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020" Oct 30 20:28:51.330328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.330058 4959 scope.go:122] "RemoveContainer" containerID="42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0" Oct 30 20:28:51.353194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.353175 4959 scope.go:122] "RemoveContainer" containerID="c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020" Oct 30 20:28:51.353483 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:51.353454 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020\": container with ID starting with c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020 not found: ID does not exist" containerID="c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020" Oct 30 20:28:51.353656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.353487 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020"} err="failed to get container status \"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020\": rpc error: code = NotFound desc = could not find container \"c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020\": container with ID starting with c794db89153fb35d9c1741818bc077bb2d674978fb3c5c9064b7331a1b054020 not found: ID does not exist" Oct 30 20:28:51.353656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.353600 4959 scope.go:122] "RemoveContainer" containerID="42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0" Oct 30 20:28:51.354048 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:51.353989 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0\": container with ID starting with 42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0 not found: ID does not exist" containerID="42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0" Oct 30 20:28:51.354048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.354023 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0"} err="failed to get container status \"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0\": rpc error: code = NotFound desc = could not find container \"42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0\": container with ID starting with 42c1247724560400453d07903e461ed618dd8b7389b083634e8716362f543ca0 not found: ID does not exist" Oct 30 20:28:51.367809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.367786 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:28:51.372237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.372214 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-30e92c-openshift-default-556c89b4cc-dgnkm"] Oct 30 20:28:51.377490 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.377471 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/618a7097-47ef-4d5b-8474-4353dface889-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.377589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.377511 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/618a7097-47ef-4d5b-8474-4353dface889-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.377589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.377527 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf8fq\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-kube-api-access-rf8fq\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.377589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.377540 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/618a7097-47ef-4d5b-8474-4353dface889-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.621838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.621804 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 20:28:51.679320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.679291 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 20:28:51.679523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.679413 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\") pod \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " Oct 30 20:28:51.679630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.679482 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\" (UniqueName: \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\") pod \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\" (UID: \"ee02872a-a046-4e3e-a6b3-8a31751cff5e\") " Oct 30 20:28:51.679863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.679836 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume" pod "ee02872a-a046-4e3e-a6b3-8a31751cff5e" (UID: "ee02872a-a046-4e3e-a6b3-8a31751cff5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:51.680101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.680078 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02872a-a046-4e3e-a6b3-8a31751cff5e-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.681862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.681835 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m" pod "ee02872a-a046-4e3e-a6b3-8a31751cff5e" (UID: "ee02872a-a046-4e3e-a6b3-8a31751cff5e"). InnerVolumeSpecName "kube-api-access-nt49m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:51.773369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.767564 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:51.781355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.781310 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\" (UniqueName: \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\") pod \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " Oct 30 20:28:51.781615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.781420 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\") pod \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\" (UID: \"f3cb41b8-04f3-4056-86cd-e674bda00fa5\") " Oct 30 20:28:51.782058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.782031 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt49m\" (UniqueName: \"kubernetes.io/projected/ee02872a-a046-4e3e-a6b3-8a31751cff5e-kube-api-access-nt49m\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.782835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.782805 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume" pod "f3cb41b8-04f3-4056-86cd-e674bda00fa5" (UID: "f3cb41b8-04f3-4056-86cd-e674bda00fa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:51.785348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.785326 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl" pod "f3cb41b8-04f3-4056-86cd-e674bda00fa5" (UID: "f3cb41b8-04f3-4056-86cd-e674bda00fa5"). InnerVolumeSpecName "kube-api-access-glhrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:51.883405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.883379 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhrl\" (UniqueName: \"kubernetes.io/projected/f3cb41b8-04f3-4056-86cd-e674bda00fa5-kube-api-access-glhrl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.883589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.883410 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3cb41b8-04f3-4056-86cd-e674bda00fa5-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:28:51.998438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:51.998405 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="618a7097-47ef-4d5b-8474-4353dface889" path="/var/lib/kubelet/pods/618a7097-47ef-4d5b-8474-4353dface889/volumes" Oct 30 20:28:52.324842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.324757 4959 generic.go:356] "Generic (PLEG): container finished" podID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerID="3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e" exitCode=0 Oct 30 20:28:52.324842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.324803 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" Oct 30 20:28:52.325057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.324850 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" event={"ID":"f3cb41b8-04f3-4056-86cd-e674bda00fa5","Type":"ContainerDied","Data":"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e"} Oct 30 20:28:52.325057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.324909 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd" event={"ID":"f3cb41b8-04f3-4056-86cd-e674bda00fa5","Type":"ContainerDied","Data":"efda0b8bdf75863ae987268f12ddb3fc6351a0b9c4ce9cd9c03a736a3950322c"} Oct 30 20:28:52.325057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.324940 4959 scope.go:122] "RemoveContainer" containerID="3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e" Oct 30 20:28:52.326422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.326366 4959 generic.go:356] "Generic (PLEG): container finished" podID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerID="e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4" exitCode=0 Oct 30 20:28:52.326422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.326409 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" Oct 30 20:28:52.326578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.326421 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" event={"ID":"ee02872a-a046-4e3e-a6b3-8a31751cff5e","Type":"ContainerDied","Data":"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4"} Oct 30 20:28:52.326578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.326467 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p" event={"ID":"ee02872a-a046-4e3e-a6b3-8a31751cff5e","Type":"ContainerDied","Data":"c9933a269556c3c1b26ce706835c843d18aaf3a21b00974c6f8892f095d15917"} Oct 30 20:28:52.336137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.336117 4959 scope.go:122] "RemoveContainer" containerID="3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e" Oct 30 20:28:52.336393 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:52.336360 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e\": container with ID starting with 3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e not found: ID does not exist" containerID="3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e" Oct 30 20:28:52.336515 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.336383 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e"} err="failed to get container status \"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e\": rpc error: code = NotFound desc = could not find container \"3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e\": container with ID starting with 3ca115f4f80fd4279b232d55b3ed51fc3a881cf91dfd8846bb7faac2cfccdc8e not found: ID does not exist" Oct 30 20:28:52.336515 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.336401 4959 scope.go:122] "RemoveContainer" containerID="e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4" Oct 30 20:28:52.345688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.345671 4959 scope.go:122] "RemoveContainer" containerID="e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4" Oct 30 20:28:52.345962 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:28:52.345943 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4\": container with ID starting with e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4 not found: ID does not exist" containerID="e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4" Oct 30 20:28:52.346030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.345972 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4"} err="failed to get container status \"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4\": rpc error: code = NotFound desc = could not find container \"e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4\": container with ID starting with e27f6757a2e6388f47434c869e2721ebabb6959dfab70b8908d8d12c0abacde4 not found: ID does not exist" Oct 30 20:28:52.374127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.374102 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 20:28:52.384755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.384732 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a165e5-68bcf45db-5w42p"] Oct 30 20:28:52.410747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.410726 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 20:28:52.413197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:52.413173 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a165e5-fb8598b5f-hknmd"] Oct 30 20:28:53.114520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.113158 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.651238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.651203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.769691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.769634 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.900585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.900550 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:53.999053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.998960 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" path="/var/lib/kubelet/pods/ee02872a-a046-4e3e-a6b3-8a31751cff5e/volumes" Oct 30 20:28:53.999379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:53.999366 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" path="/var/lib/kubelet/pods/f3cb41b8-04f3-4056-86cd-e674bda00fa5/volumes" Oct 30 20:28:54.296298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:54.296221 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:55.711346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:55.711306 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:56.720969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:56.720940 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 20:28:56.771477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:56.771445 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 20:28:56.935034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:56.935003 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:57.052273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:57.052188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:57.487166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:57.487130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:59.120115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:28:59.120082 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:00.787644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:00.787600 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:00.892600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:00.892563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:00.994148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:00.994115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:01.158194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:01.158152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:01.721570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:01.721524 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" containerID="cri-o://738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a" gracePeriod=25 Oct 30 20:29:01.772253 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:01.772216 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" containerID="cri-o://1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0" gracePeriod=25 Oct 30 20:29:02.156290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:02.156257 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:02.156801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:02.156757 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:02.809377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:02.809322 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:04.549116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:04.549074 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:06.323921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:06.323867 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:08.147654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:08.147612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:10.273646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:10.273610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.072320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:11.072288 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.328949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:11.328868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.441386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:11.441353 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:11.684201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:11.684168 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:12.155675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:12.155642 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:12.157095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:12.157061 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:12.159961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:12.159932 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:29:12.264566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:12.264530 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:13.205998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:13.205961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:13.818328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:13.818300 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:13.924859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:13.924820 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:15.029509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:15.029468 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:16.066096 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:29:16.066065 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:29:16.743018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:16.742985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.127815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:18.127786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.239899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:18.239865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.458931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:18.458887 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:18.622736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:18.622702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:20.634811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:20.634775 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:22.047329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.047305 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 20:29:22.088325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.088302 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 20:29:22.119210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.119179 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\" (UniqueName: \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\") pod \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " Oct 30 20:29:22.119385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.119257 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\") pod \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\" (UID: \"3e53623f-9873-45ad-bb50-e7dea4b10f2a\") " Oct 30 20:29:22.120100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.119833 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume" pod "3e53623f-9873-45ad-bb50-e7dea4b10f2a" (UID: "3e53623f-9873-45ad-bb50-e7dea4b10f2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:22.121645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.121621 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs" pod "3e53623f-9873-45ad-bb50-e7dea4b10f2a" (UID: "3e53623f-9873-45ad-bb50-e7dea4b10f2a"). InnerVolumeSpecName "kube-api-access-pdkxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:22.220793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.220761 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\" (UniqueName: \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\") pod \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " Oct 30 20:29:22.220977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.220804 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\") pod \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\" (UID: \"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2\") " Oct 30 20:29:22.221201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.221186 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdkxs\" (UniqueName: \"kubernetes.io/projected/3e53623f-9873-45ad-bb50-e7dea4b10f2a-kube-api-access-pdkxs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:22.221263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.221204 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e53623f-9873-45ad-bb50-e7dea4b10f2a-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:22.221263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.221250 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume" pod "2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" (UID: "2d3b038e-72a5-43d9-985e-9e03ae2d8fe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:22.222972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.222955 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td" pod "2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" (UID: "2d3b038e-72a5-43d9-985e-9e03ae2d8fe2"). InnerVolumeSpecName "kube-api-access-8s6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:22.322085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.322011 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6td\" (UniqueName: \"kubernetes.io/projected/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-kube-api-access-8s6td\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:22.322085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.322039 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:22.520968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.520929 4959 generic.go:356] "Generic (PLEG): container finished" podID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerID="1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0" exitCode=0 Oct 30 20:29:22.521149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.520979 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" Oct 30 20:29:22.521149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.521012 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" event={"ID":"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2","Type":"ContainerDied","Data":"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0"} Oct 30 20:29:22.521149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.521064 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59" event={"ID":"2d3b038e-72a5-43d9-985e-9e03ae2d8fe2","Type":"ContainerDied","Data":"218f10e0f47136ec60ce3d0913b4b1c85c2a563edc603fcd8ce96d5fef6da5e5"} Oct 30 20:29:22.521149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.521088 4959 scope.go:122] "RemoveContainer" containerID="1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0" Oct 30 20:29:22.522671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.522648 4959 generic.go:356] "Generic (PLEG): container finished" podID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerID="738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a" exitCode=0 Oct 30 20:29:22.522796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.522690 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" event={"ID":"3e53623f-9873-45ad-bb50-e7dea4b10f2a","Type":"ContainerDied","Data":"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a"} Oct 30 20:29:22.522796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.522720 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" event={"ID":"3e53623f-9873-45ad-bb50-e7dea4b10f2a","Type":"ContainerDied","Data":"e653cfbd54e586f4613fde2154d0ca752f22081ece487c116841ef45df833758"} Oct 30 20:29:22.522890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.522805 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv" Oct 30 20:29:22.536214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.536192 4959 scope.go:122] "RemoveContainer" containerID="1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0" Oct 30 20:29:22.536647 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:29:22.536624 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0\": container with ID starting with 1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0 not found: ID does not exist" containerID="1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0" Oct 30 20:29:22.536771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.536654 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0"} err="failed to get container status \"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0\": rpc error: code = NotFound desc = could not find container \"1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0\": container with ID starting with 1a9c9ae8db8782afa709f918347c872be92ebf19d3544445f616f30d913ebbd0 not found: ID does not exist" Oct 30 20:29:22.536771 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.536677 4959 scope.go:122] "RemoveContainer" containerID="738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a" Oct 30 20:29:22.546290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.546274 4959 scope.go:122] "RemoveContainer" containerID="738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a" Oct 30 20:29:22.546625 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:29:22.546608 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a\": container with ID starting with 738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a not found: ID does not exist" containerID="738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a" Oct 30 20:29:22.546701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.546630 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a"} err="failed to get container status \"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a\": rpc error: code = NotFound desc = could not find container \"738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a\": container with ID starting with 738eac599b20b6c28c6a4614f3b664c5c01171b22ff82f43a24157978daaa16a not found: ID does not exist" Oct 30 20:29:22.572590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.572533 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 20:29:22.586485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.586457 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c8fa69-595cc9cb7-rdw59"] Oct 30 20:29:22.630523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.630471 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 20:29:22.634727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.634703 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c8fa69-7ff7dbd956-xphtv"] Oct 30 20:29:22.920175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:22.920144 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:23.998188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:23.998142 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" path="/var/lib/kubelet/pods/2d3b038e-72a5-43d9-985e-9e03ae2d8fe2/volumes" Oct 30 20:29:23.998699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:23.998683 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" path="/var/lib/kubelet/pods/3e53623f-9873-45ad-bb50-e7dea4b10f2a/volumes" Oct 30 20:29:24.559200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:24.559170 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:26.166561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:26.166522 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:27.910572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:27.910536 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:29.378357 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:29.378319 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:29.496947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:29.496908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:29.648060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:29.648031 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:31.180427 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:31.180398 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:31.828446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:31.828406 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:31.975521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:31.972636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:32.083812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:32.083725 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:32.446835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:32.446792 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:33.681114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:33.681082 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:33.921714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:33.921670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:34.174676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:34.174640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:35.710056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:35.710011 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:36.512273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:36.512248 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:29:36.519614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:36.519593 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:29:37.217309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.217273 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:29:37.217811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.217482 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" podUID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" containerName="maas-api" containerID="cri-o://b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847" gracePeriod=30 Oct 30 20:29:37.287759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.287690 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:37.509144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.509061 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:37.549128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.549105 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:29:37.627933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.627897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:37.644810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.644771 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\" (UniqueName: \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\") pod \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " Oct 30 20:29:37.645003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.644868 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\") pod \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " Oct 30 20:29:37.645003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.644989 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") pod \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " Oct 30 20:29:37.645118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.645063 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\") pod \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\" (UID: \"cda9f9c3-c416-471b-9dcb-76bac5837aaf\") " Oct 30 20:29:37.646396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.646285 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca" pod "cda9f9c3-c416-471b-9dcb-76bac5837aaf" (UID: "cda9f9c3-c416-471b-9dcb-76bac5837aaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:29:37.648222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.648194 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs" pod "cda9f9c3-c416-471b-9dcb-76bac5837aaf" (UID: "cda9f9c3-c416-471b-9dcb-76bac5837aaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:37.648222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.648211 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7" pod "cda9f9c3-c416-471b-9dcb-76bac5837aaf" (UID: "cda9f9c3-c416-471b-9dcb-76bac5837aaf"). InnerVolumeSpecName "kube-api-access-9z4b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:29:37.648988 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.648964 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls" pod "cda9f9c3-c416-471b-9dcb-76bac5837aaf" (UID: "cda9f9c3-c416-471b-9dcb-76bac5837aaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:29:37.653048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.653012 4959 generic.go:356] "Generic (PLEG): container finished" podID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" containerID="b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847" exitCode=0 Oct 30 20:29:37.653166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.653088 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" Oct 30 20:29:37.653166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.653095 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" event={"ID":"cda9f9c3-c416-471b-9dcb-76bac5837aaf","Type":"ContainerDied","Data":"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847"} Oct 30 20:29:37.653285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.653196 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq" event={"ID":"cda9f9c3-c416-471b-9dcb-76bac5837aaf","Type":"ContainerDied","Data":"002cf805239e7f6a3b11c4420d28846f5ea97275756e2688ab6656732d3f2895"} Oct 30 20:29:37.653285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.653226 4959 scope.go:122] "RemoveContainer" containerID="b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847" Oct 30 20:29:37.679075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.679057 4959 scope.go:122] "RemoveContainer" containerID="b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847" Oct 30 20:29:37.679379 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:29:37.679359 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847\": container with ID starting with b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847 not found: ID does not exist" containerID="b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847" Oct 30 20:29:37.679430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.679388 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847"} err="failed to get container status \"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847\": rpc error: code = NotFound desc = could not find container \"b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847\": container with ID starting with b4614eb34b3f8004b9edc150d0bf6394ee6cede42db298a21d3d3827cf740847 not found: ID does not exist" Oct 30 20:29:37.716043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.716012 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:29:37.719528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.719482 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c19563-5d5599b985-bnxmq"] Oct 30 20:29:37.764583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.754381 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.764583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.754417 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cda9f9c3-c416-471b-9dcb-76bac5837aaf-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.764583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.754431 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cda9f9c3-c416-471b-9dcb-76bac5837aaf-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.764583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.754445 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4b7\" (UniqueName: \"kubernetes.io/projected/cda9f9c3-c416-471b-9dcb-76bac5837aaf-kube-api-access-9z4b7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:29:37.959828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.959791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:37.996551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:37.996521 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" path="/var/lib/kubelet/pods/cda9f9c3-c416-471b-9dcb-76bac5837aaf/volumes" Oct 30 20:29:38.083844 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:38.083759 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:38.201863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:38.201836 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:38.635882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:38.635841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:39.380897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:39.380868 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:39.488214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:39.488179 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:39.729414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:39.729384 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:40.199832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:40.199797 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:40.313903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:40.313866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:42.197316 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:42.197285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:43.798979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:43.797195 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:44.465878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:44.465849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:44.610821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:44.610784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:44.746571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:44.746446 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:45.138190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:45.138154 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:45.250757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:45.250725 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 20:29:45.403352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:45.403267 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:46.067119 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:29:46.067086 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:29:46.884623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:46.884588 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:47.828874 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:47.828845 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:47.948366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:47.948333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:48.217651 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:48.217620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:49.777647 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:49.777603 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:50.668683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:50.668648 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 20:29:50.794430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:50.794393 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:50.913521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:50.909479 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:51.840717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:51.840686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:52.381878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:52.381840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:52.513103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:52.513052 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:52.765614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:52.765578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:53.247160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:53.247120 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:53.987982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:53.987947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:55.669914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:55.669874 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" containerID="cri-o://ece60ccd8c6bb99fb191b6af8f73697bc478351f1554e1348cc6b2150d08f7a1" gracePeriod=25 Oct 30 20:29:55.816481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:55.816442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:57.624668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:57.624635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:58.209664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:58.209637 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 20:29:58.222424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:58.222392 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 20:29:58.347843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:58.347804 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:59.799448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:29:59.799410 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:00.172150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.172114 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:00.173227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173192 4959 cpu_manager.go:414] "containerMap: removing container" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" Oct 30 20:30:00.173227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173224 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173237 4959 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" containerName="payload-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173246 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" containerName="payload-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173258 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173265 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173279 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173287 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173317 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f1bde44b-797a-4bb4-855e-c23e21257189" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173323 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bde44b-797a-4bb4-855e-c23e21257189" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173339 4959 cpu_manager.go:414] "containerMap: removing container" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173346 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173376 4959 cpu_manager.go:414] "containerMap: removing container" podUID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" containerName="maas-api" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173383 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" containerName="maas-api" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173395 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173400 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173409 4959 cpu_manager.go:414] "containerMap: removing container" podUID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" containerName="maas-api" Oct 30 20:30:00.173409 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173413 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173420 4959 cpu_manager.go:414] "containerMap: removing container" podUID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173425 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173431 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173435 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173585 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c9f14a-fae3-4dc8-bfba-edb439bd763a" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173594 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda9f9c3-c416-471b-9dcb-76bac5837aaf" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173599 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee02872a-a046-4e3e-a6b3-8a31751cff5e" containerName="payload-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173609 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab9c04d-8cdc-4dd5-a148-1e68991f92cc" containerName="istio-proxy" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173614 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd9b57-b4e9-4eaa-8819-4cdb8e27a5cb" containerName="payload-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173621 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e53623f-9873-45ad-bb50-e7dea4b10f2a" containerName="payload-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173626 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cb41b8-04f3-4056-86cd-e674bda00fa5" containerName="payload-pre-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173634 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bde44b-797a-4bb4-855e-c23e21257189" containerName="payload-pre-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173640 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173647 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="345f6b5a-1ce0-4e7b-8bf5-1eae3e6edd3d" containerName="maas-api" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173652 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3b038e-72a5-43d9-985e-9e03ae2d8fe2" containerName="payload-pre-processing" Oct 30 20:30:00.174091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.173659 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.178783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.178766 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.198581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.198556 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:30:00.213199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.213150 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:00.214108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.214080 4959 cpu_manager.go:414] "containerMap: removing container" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.214250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.214110 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="618a7097-47ef-4d5b-8474-4353dface889" containerName="istio-proxy" Oct 30 20:30:00.218754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.218728 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:00.218881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.218860 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.219107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.219089 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:00.223217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.223198 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.237466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.237440 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:00.245136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.245104 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhj5v\" (UniqueName: \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.247193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.247162 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.247312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.247224 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.247312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.247282 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwds\" (UniqueName: \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.247397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.247368 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.247458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.247439 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvs5\" (UniqueName: \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.249721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.249698 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:00.254750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.254730 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:00.254884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.254864 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.272488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.272454 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" containerID="cri-o://ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee" gracePeriod=45 Oct 30 20:30:00.284882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.284678 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:00.348911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.348878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhj5v\" (UniqueName: \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.349096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.348922 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.349096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.348943 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.349096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.348984 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4plj\" (UniqueName: \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.349096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349023 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwds\" (UniqueName: \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.349352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349152 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.349352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349189 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.349352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349307 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvs5\" (UniqueName: \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.349658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349638 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.349717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349658 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.349802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.349785 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.373107 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.373071 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvs5\" (UniqueName: \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\") pod \"maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.373267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.373121 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhj5v\" (UniqueName: \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\") pod \"maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.373267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.373197 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwds\" (UniqueName: \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.450735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.450695 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4plj\" (UniqueName: \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.450888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.450825 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.451696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.451667 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.472883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.472853 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4plj\" (UniqueName: \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.492536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.492487 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:00.532607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.532581 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 20:30:00.537175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.537152 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:00.552139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.550184 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:00.553037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.552998 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\" (UniqueName: \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553114 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553190 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553267 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553383 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\" (UniqueName: \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553579 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\") pod \"33e13789-931c-44a4-bbeb-60d874d39502\" (UID: \"33e13789-931c-44a4-bbeb-60d874d39502\") " Oct 30 20:30:00.553617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553593 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.553953 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553654 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.553953 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.553884 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.554870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.554583 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.554870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.554607 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.554870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.554619 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.556338 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.556317 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:00.557907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.557883 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "kube-api-access-pmrhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:00.559893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.559861 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm" pod "33e13789-931c-44a4-bbeb-60d874d39502" (UID: "33e13789-931c-44a4-bbeb-60d874d39502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:00.572826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.572796 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:00.656337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.656267 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrhv\" (UniqueName: \"kubernetes.io/projected/33e13789-931c-44a4-bbeb-60d874d39502-kube-api-access-pmrhv\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.656337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.656300 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e13789-931c-44a4-bbeb-60d874d39502-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.656337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.656316 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e13789-931c-44a4-bbeb-60d874d39502-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:00.803015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.802979 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:00.807893 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:30:00.807859 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cbe136_f3d8_431c_a9a6_9c970fb258c0.slice/crio-78cb95a677cf210cf410c397fc72c2a9905f4be5c33966d5369e44ee1124c0a7 WatchSource:0}: Error finding container 78cb95a677cf210cf410c397fc72c2a9905f4be5c33966d5369e44ee1124c0a7: Status 404 returned error can't find the container with id 78cb95a677cf210cf410c397fc72c2a9905f4be5c33966d5369e44ee1124c0a7 Oct 30 20:30:00.815418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.815392 4959 generic.go:356] "Generic (PLEG): container finished" podID="33e13789-931c-44a4-bbeb-60d874d39502" containerID="ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee" exitCode=0 Oct 30 20:30:00.815549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.815429 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" Oct 30 20:30:00.815549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.815459 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" event={"ID":"33e13789-931c-44a4-bbeb-60d874d39502","Type":"ContainerDied","Data":"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee"} Oct 30 20:30:00.815549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.815537 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk" event={"ID":"33e13789-931c-44a4-bbeb-60d874d39502","Type":"ContainerDied","Data":"e5317e40a7920ed6503d178bf7e921f0a613bbc9cb1c88427f493d25c65506a7"} Oct 30 20:30:00.815717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.815568 4959 scope.go:122] "RemoveContainer" containerID="ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee" Oct 30 20:30:00.831616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.831589 4959 scope.go:122] "RemoveContainer" containerID="ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee" Oct 30 20:30:00.832020 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:00.831994 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee\": container with ID starting with ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee not found: ID does not exist" containerID="ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee" Oct 30 20:30:00.832140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.832030 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee"} err="failed to get container status \"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee\": rpc error: code = NotFound desc = could not find container \"ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee\": container with ID starting with ac7c0ae9c36934bfbeee53a36be0784e3ac46c47dc533827537f2312ff0d22ee not found: ID does not exist" Oct 30 20:30:00.895325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.895293 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 20:30:00.897886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.897860 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:00.899389 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:30:00.899364 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a0b3fe_00dd_4085_a295_822662873fc6.slice/crio-4174bc021ce57470b625cfa450672ec884fbb3f42e1bcc6a39a8c6a16792791c WatchSource:0}: Error finding container 4174bc021ce57470b625cfa450672ec884fbb3f42e1bcc6a39a8c6a16792791c: Status 404 returned error can't find the container with id 4174bc021ce57470b625cfa450672ec884fbb3f42e1bcc6a39a8c6a16792791c Oct 30 20:30:00.899615 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:30:00.899590 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88143ed0_d601_4e9e_8086_65378962508b.slice/crio-1f63a4ef5ec60703daa1e9eaa33f2f519a6b07569b444508c63d3b8f9903a124 WatchSource:0}: Error finding container 1f63a4ef5ec60703daa1e9eaa33f2f519a6b07569b444508c63d3b8f9903a124: Status 404 returned error can't find the container with id 1f63a4ef5ec60703daa1e9eaa33f2f519a6b07569b444508c63d3b8f9903a124 Oct 30 20:30:00.900990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.900972 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:00.903314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.903295 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-284bf6/distinct-w2-284bf6-kserve-c9cb575d7-9gblk"] Oct 30 20:30:00.939304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:00.939273 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:00.943735 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:30:00.943697 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150b05b0_19bd_4138_bf01_c5f4e103257e.slice/crio-d57720fe5b2228c37ff9c7f8aa872782c7d79a44529233d91fb00db8429a3897 WatchSource:0}: Error finding container d57720fe5b2228c37ff9c7f8aa872782c7d79a44529233d91fb00db8429a3897: Status 404 returned error can't find the container with id d57720fe5b2228c37ff9c7f8aa872782c7d79a44529233d91fb00db8429a3897 Oct 30 20:30:01.831681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.831639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:01.836645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.836615 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerStarted","Data":"345685dfcf13059cdb74ab96981f4bf134e1314177049045f896378f56bda35b"} Oct 30 20:30:01.836806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.836697 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerStarted","Data":"4174bc021ce57470b625cfa450672ec884fbb3f42e1bcc6a39a8c6a16792791c"} Oct 30 20:30:01.838324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.838295 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerStarted","Data":"b5f472528a7ccaef74c2e41b4025cdd5d5d48a4ee1da3c87044982af3e5453cd"} Oct 30 20:30:01.838438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.838347 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerStarted","Data":"1f63a4ef5ec60703daa1e9eaa33f2f519a6b07569b444508c63d3b8f9903a124"} Oct 30 20:30:01.839872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.839847 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerStarted","Data":"90b29725ee3afb3b9afbda8cd033d2e65d0b03886730200fcd7c2ec837cd92fc"} Oct 30 20:30:01.839981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.839892 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerStarted","Data":"d57720fe5b2228c37ff9c7f8aa872782c7d79a44529233d91fb00db8429a3897"} Oct 30 20:30:01.841232 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.841210 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerStarted","Data":"f12f2c6ae1e3d8d5bc949f3c0e755a19cb43085d2e80566927dd8cb739c538e6"} Oct 30 20:30:01.841366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.841252 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerStarted","Data":"78cb95a677cf210cf410c397fc72c2a9905f4be5c33966d5369e44ee1124c0a7"} Oct 30 20:30:01.877630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.877576 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" podStartSLOduration=1.877479556 podStartE2EDuration="1.877479556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.876975769 +0000 UTC m=+3326.006209693" watchObservedRunningTime="2026-10-30 20:30:01.877479556 +0000 UTC m=+3326.006713470" Oct 30 20:30:01.934270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.933535 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" podStartSLOduration=1.933514877 podStartE2EDuration="1.933514877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.930188531 +0000 UTC m=+3326.059422475" watchObservedRunningTime="2026-10-30 20:30:01.933514877 +0000 UTC m=+3326.062748796" Oct 30 20:30:01.970207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.970144 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" podStartSLOduration=1.970125333 podStartE2EDuration="1.970125333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.966279876 +0000 UTC m=+3326.095513810" watchObservedRunningTime="2026-10-30 20:30:01.970125333 +0000 UTC m=+3326.099359257" Oct 30 20:30:01.999003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:01.998974 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e13789-931c-44a4-bbeb-60d874d39502" path="/var/lib/kubelet/pods/33e13789-931c-44a4-bbeb-60d874d39502/volumes" Oct 30 20:30:02.003351 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.003312 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" podStartSLOduration=2.003299722 podStartE2EDuration="2.003299722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:02.002942108 +0000 UTC m=+3326.132176029" watchObservedRunningTime="2026-10-30 20:30:02.003299722 +0000 UTC m=+3326.132533643" Oct 30 20:30:02.851967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.851930 4959 generic.go:356] "Generic (PLEG): container finished" podID="95a0b3fe-00dd-4085-a295-822662873fc6" containerID="345685dfcf13059cdb74ab96981f4bf134e1314177049045f896378f56bda35b" exitCode=60 Oct 30 20:30:02.852419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.852013 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerDied","Data":"345685dfcf13059cdb74ab96981f4bf134e1314177049045f896378f56bda35b"} Oct 30 20:30:02.852419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.852297 4959 scope.go:122] "RemoveContainer" containerID="345685dfcf13059cdb74ab96981f4bf134e1314177049045f896378f56bda35b" Oct 30 20:30:02.853809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.853784 4959 generic.go:356] "Generic (PLEG): container finished" podID="88143ed0-d601-4e9e-8086-65378962508b" containerID="b5f472528a7ccaef74c2e41b4025cdd5d5d48a4ee1da3c87044982af3e5453cd" exitCode=60 Oct 30 20:30:02.853919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.853867 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerDied","Data":"b5f472528a7ccaef74c2e41b4025cdd5d5d48a4ee1da3c87044982af3e5453cd"} Oct 30 20:30:02.854117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.854102 4959 scope.go:122] "RemoveContainer" containerID="b5f472528a7ccaef74c2e41b4025cdd5d5d48a4ee1da3c87044982af3e5453cd" Oct 30 20:30:02.855717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.855668 4959 generic.go:356] "Generic (PLEG): container finished" podID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerID="90b29725ee3afb3b9afbda8cd033d2e65d0b03886730200fcd7c2ec837cd92fc" exitCode=60 Oct 30 20:30:02.855781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.855749 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerDied","Data":"90b29725ee3afb3b9afbda8cd033d2e65d0b03886730200fcd7c2ec837cd92fc"} Oct 30 20:30:02.855998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.855983 4959 scope.go:122] "RemoveContainer" containerID="90b29725ee3afb3b9afbda8cd033d2e65d0b03886730200fcd7c2ec837cd92fc" Oct 30 20:30:02.857588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.857569 4959 generic.go:356] "Generic (PLEG): container finished" podID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerID="f12f2c6ae1e3d8d5bc949f3c0e755a19cb43085d2e80566927dd8cb739c538e6" exitCode=60 Oct 30 20:30:02.857660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.857643 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerDied","Data":"f12f2c6ae1e3d8d5bc949f3c0e755a19cb43085d2e80566927dd8cb739c538e6"} Oct 30 20:30:02.857836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:02.857823 4959 scope.go:122] "RemoveContainer" containerID="f12f2c6ae1e3d8d5bc949f3c0e755a19cb43085d2e80566927dd8cb739c538e6" Oct 30 20:30:03.857393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.857361 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:03.866910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.866883 4959 generic.go:356] "Generic (PLEG): container finished" podID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerID="0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20" exitCode=60 Oct 30 20:30:03.867048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.866967 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerDied","Data":"0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20"} Oct 30 20:30:03.867113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.867055 4959 scope.go:122] "RemoveContainer" containerID="90b29725ee3afb3b9afbda8cd033d2e65d0b03886730200fcd7c2ec837cd92fc" Oct 30 20:30:03.867391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.867372 4959 scope.go:122] "RemoveContainer" containerID="0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20" Oct 30 20:30:03.867789 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:03.867764 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp_odh-ai-gateway-infra(150b05b0-19bd-4138-bf01-c5f4e103257e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" Oct 30 20:30:03.869639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.869615 4959 generic.go:356] "Generic (PLEG): container finished" podID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerID="ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d" exitCode=60 Oct 30 20:30:03.869833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.869686 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerDied","Data":"ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d"} Oct 30 20:30:03.870020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.870000 4959 scope.go:122] "RemoveContainer" containerID="ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d" Oct 30 20:30:03.870374 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:03.870338 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7_odh-ai-gateway-infra(e4cbe136-f3d8-431c-a9a6-9c970fb258c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" Oct 30 20:30:03.871982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.871956 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerStarted","Data":"ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2"} Oct 30 20:30:03.874274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.874249 4959 generic.go:356] "Generic (PLEG): container finished" podID="88143ed0-d601-4e9e-8086-65378962508b" containerID="a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75" exitCode=60 Oct 30 20:30:03.874381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.874288 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerDied","Data":"a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75"} Oct 30 20:30:03.874558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:03.874543 4959 scope.go:122] "RemoveContainer" containerID="a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75" Oct 30 20:30:03.874880 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:03.874843 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4_odh-ai-gateway-infra(88143ed0-d601-4e9e-8086-65378962508b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" podUID="88143ed0-d601-4e9e-8086-65378962508b" Oct 30 20:30:04.023899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.023874 4959 scope.go:122] "RemoveContainer" containerID="f12f2c6ae1e3d8d5bc949f3c0e755a19cb43085d2e80566927dd8cb739c538e6" Oct 30 20:30:04.045368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.044848 4959 scope.go:122] "RemoveContainer" containerID="b5f472528a7ccaef74c2e41b4025cdd5d5d48a4ee1da3c87044982af3e5453cd" Oct 30 20:30:04.097001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.096975 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:04.893397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.893364 4959 scope.go:122] "RemoveContainer" containerID="a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75" Oct 30 20:30:04.893916 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:04.893760 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4_odh-ai-gateway-infra(88143ed0-d601-4e9e-8086-65378962508b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" podUID="88143ed0-d601-4e9e-8086-65378962508b" Oct 30 20:30:04.895101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.895076 4959 scope.go:122] "RemoveContainer" containerID="0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20" Oct 30 20:30:04.895393 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:04.895364 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp_odh-ai-gateway-infra(150b05b0-19bd-4138-bf01-c5f4e103257e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" Oct 30 20:30:04.896683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.896664 4959 scope.go:122] "RemoveContainer" containerID="ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d" Oct 30 20:30:04.896966 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:04.896936 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7_odh-ai-gateway-infra(e4cbe136-f3d8-431c-a9a6-9c970fb258c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" Oct 30 20:30:04.898318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.898297 4959 generic.go:356] "Generic (PLEG): container finished" podID="95a0b3fe-00dd-4085-a295-822662873fc6" containerID="ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2" exitCode=60 Oct 30 20:30:04.898413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.898329 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerDied","Data":"ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2"} Oct 30 20:30:04.898413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.898393 4959 scope.go:122] "RemoveContainer" containerID="345685dfcf13059cdb74ab96981f4bf134e1314177049045f896378f56bda35b" Oct 30 20:30:04.898642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:04.898621 4959 scope.go:122] "RemoveContainer" containerID="ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2" Oct 30 20:30:04.898886 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:04.898867 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5_odh-ai-gateway-infra(95a0b3fe-00dd-4085-a295-822662873fc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" Oct 30 20:30:05.087470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:05.087431 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:05.907489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:05.907467 4959 scope.go:122] "RemoveContainer" containerID="ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2" Oct 30 20:30:05.907911 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:05.907776 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5_odh-ai-gateway-infra(95a0b3fe-00dd-4085-a295-822662873fc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" Oct 30 20:30:06.272020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:06.271944 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:08.243721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:08.243687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:08.450976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:08.450940 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:08.934304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:08.934274 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:10.337328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:10.337292 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:11.850610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:11.850577 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.066821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:12.066786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.175918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:12.175879 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.551095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:12.551004 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.743124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:12.743093 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:12.858426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:12.858346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:13.231692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.231641 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" containerID="cri-o://25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293" gracePeriod=45 Oct 30 20:30:13.249958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.249923 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" containerID="cri-o://c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243" gracePeriod=45 Oct 30 20:30:13.254159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.254129 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": EOF" Oct 30 20:30:13.283156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.283119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:13.526305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.526275 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 20:30:13.559633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.559609 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 20:30:13.636917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.636885 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\" (UniqueName: \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.636930 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\" (UniqueName: \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.636983 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637017 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637077 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\" (UniqueName: \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637129 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637230 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637449 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\" (UniqueName: \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\") pod \"36b62495-d391-4c86-b5c6-0f5989657f27\" (UID: \"36b62495-d391-4c86-b5c6-0f5989657f27\") " Oct 30 20:30:13.637590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637549 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637594 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637673 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637712 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\") pod \"928c98ce-c4d5-417d-a715-e92d39072d88\" (UID: \"928c98ce-c4d5-417d-a715-e92d39072d88\") " Oct 30 20:30:13.637995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.637937 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.638193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.638171 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.638410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.638388 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.638527 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.638415 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.638671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.638648 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.638878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.638855 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.639066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.639047 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.639151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.639132 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.639984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.639947 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "kube-api-access-hkhrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:13.640092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.640034 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:13.640092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.640056 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.640991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.640954 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:13.641188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.641151 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs" pod "928c98ce-c4d5-417d-a715-e92d39072d88" (UID: "928c98ce-c4d5-417d-a715-e92d39072d88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:13.641544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.641521 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l" pod "36b62495-d391-4c86-b5c6-0f5989657f27" (UID: "36b62495-d391-4c86-b5c6-0f5989657f27"). InnerVolumeSpecName "kube-api-access-zv94l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:13.739445 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739413 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739445 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739439 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b62495-d391-4c86-b5c6-0f5989657f27-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739445 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739447 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739457 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739464 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739471 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/928c98ce-c4d5-417d-a715-e92d39072d88-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739479 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhrf\" (UniqueName: \"kubernetes.io/projected/928c98ce-c4d5-417d-a715-e92d39072d88-kube-api-access-hkhrf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739487 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv94l\" (UniqueName: \"kubernetes.io/projected/36b62495-d391-4c86-b5c6-0f5989657f27-kube-api-access-zv94l\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739531 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b62495-d391-4c86-b5c6-0f5989657f27-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.739682 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.739541 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/928c98ce-c4d5-417d-a715-e92d39072d88-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:13.965584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.965545 4959 generic.go:356] "Generic (PLEG): container finished" podID="928c98ce-c4d5-417d-a715-e92d39072d88" containerID="25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293" exitCode=0 Oct 30 20:30:13.966073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.965612 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" Oct 30 20:30:13.966073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.965625 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" event={"ID":"928c98ce-c4d5-417d-a715-e92d39072d88","Type":"ContainerDied","Data":"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293"} Oct 30 20:30:13.966073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.965671 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b" event={"ID":"928c98ce-c4d5-417d-a715-e92d39072d88","Type":"ContainerDied","Data":"149aee50fad615043edd9388cd064f98aaf6dde9e957010a1de4e74f2522f23d"} Oct 30 20:30:13.966073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.965688 4959 scope.go:122] "RemoveContainer" containerID="25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293" Oct 30 20:30:13.967688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.967658 4959 generic.go:356] "Generic (PLEG): container finished" podID="36b62495-d391-4c86-b5c6-0f5989657f27" containerID="c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243" exitCode=0 Oct 30 20:30:13.967811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.967704 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" Oct 30 20:30:13.967811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.967732 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" event={"ID":"36b62495-d391-4c86-b5c6-0f5989657f27","Type":"ContainerDied","Data":"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243"} Oct 30 20:30:13.967811 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.967766 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w" event={"ID":"36b62495-d391-4c86-b5c6-0f5989657f27","Type":"ContainerDied","Data":"f64ce8871c6dc821d381cae9652c77bb85ca65e1ac4375e620183a42890ade59"} Oct 30 20:30:13.978211 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.978192 4959 scope.go:122] "RemoveContainer" containerID="25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293" Oct 30 20:30:13.978479 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:13.978459 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293\": container with ID starting with 25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293 not found: ID does not exist" containerID="25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293" Oct 30 20:30:13.978607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.978487 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293"} err="failed to get container status \"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293\": rpc error: code = NotFound desc = could not find container \"25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293\": container with ID starting with 25fa3e2298aef7c3cae284871f8a6cd3b1a074ceaacdfef63d57fc9534935293 not found: ID does not exist" Oct 30 20:30:13.978607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.978538 4959 scope.go:122] "RemoveContainer" containerID="c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243" Oct 30 20:30:13.988189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.988172 4959 scope.go:122] "RemoveContainer" containerID="c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243" Oct 30 20:30:13.988458 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:13.988434 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243\": container with ID starting with c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243 not found: ID does not exist" containerID="c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243" Oct 30 20:30:13.988567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:13.988468 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243"} err="failed to get container status \"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243\": rpc error: code = NotFound desc = could not find container \"c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243\": container with ID starting with c897ddec9a85abe6e02df1aadeb611e32397e122da337a817f583f146f54f243 not found: ID does not exist" Oct 30 20:30:14.019907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.019884 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 20:30:14.030583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.030562 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/premium-w3-01f7d3-kserve-57b97b98cb-qpq5w"] Oct 30 20:30:14.065969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.065945 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 20:30:14.071074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.071053 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-01f7d3/distinct-w3-01f7d3-kserve-755b65cb59-dmg4b"] Oct 30 20:30:14.995472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.994715 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:14.996700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.996621 4959 scope.go:122] "RemoveContainer" containerID="0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20" Oct 30 20:30:14.997954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:14.997216 4959 scope.go:122] "RemoveContainer" containerID="ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d" Oct 30 20:30:15.087613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:15.087583 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:16.017537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.017463 4959 generic.go:356] "Generic (PLEG): container finished" podID="d3a8512e-1144-4d81-a115-51de804273cc" containerID="ece60ccd8c6bb99fb191b6af8f73697bc478351f1554e1348cc6b2150d08f7a1" exitCode=0 Oct 30 20:30:16.017996 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.017965 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" path="/var/lib/kubelet/pods/36b62495-d391-4c86-b5c6-0f5989657f27/volumes" Oct 30 20:30:16.018639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.018616 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" path="/var/lib/kubelet/pods/928c98ce-c4d5-417d-a715-e92d39072d88/volumes" Oct 30 20:30:16.019023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.019003 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" event={"ID":"d3a8512e-1144-4d81-a115-51de804273cc","Type":"ContainerDied","Data":"ece60ccd8c6bb99fb191b6af8f73697bc478351f1554e1348cc6b2150d08f7a1"} Oct 30 20:30:16.020061 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.020037 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerStarted","Data":"57e0bac9948b1b2cb564085927bed0d7dbf66722352a00d8767465bc29f3e14c"} Oct 30 20:30:16.023304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.023273 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerStarted","Data":"09c865ff775df9b397a5ced13a25ce83dfbe05c3ac659a186398980de377380e"} Oct 30 20:30:16.044572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.044542 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 20:30:16.080210 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:16.080150 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:30:16.182914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.170332 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\" (UniqueName: \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\") pod \"d3a8512e-1144-4d81-a115-51de804273cc\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " Oct 30 20:30:16.182914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.170412 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\") pod \"d3a8512e-1144-4d81-a115-51de804273cc\" (UID: \"d3a8512e-1144-4d81-a115-51de804273cc\") " Oct 30 20:30:16.182914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.175675 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume" pod "d3a8512e-1144-4d81-a115-51de804273cc" (UID: "d3a8512e-1144-4d81-a115-51de804273cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:16.182914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.179810 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl" pod "d3a8512e-1144-4d81-a115-51de804273cc" (UID: "d3a8512e-1144-4d81-a115-51de804273cc"). InnerVolumeSpecName "kube-api-access-c9wnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:16.274398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.274356 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wnl\" (UniqueName: \"kubernetes.io/projected/d3a8512e-1144-4d81-a115-51de804273cc-kube-api-access-c9wnl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:16.274398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:16.274399 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a8512e-1144-4d81-a115-51de804273cc-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:17.035192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.035161 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" Oct 30 20:30:17.035192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.035168 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx" event={"ID":"d3a8512e-1144-4d81-a115-51de804273cc","Type":"ContainerDied","Data":"15f9822c5379601f1324f74beb0b9f5706da0591e60abb082f90f580f33b3821"} Oct 30 20:30:17.035752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.035233 4959 scope.go:122] "RemoveContainer" containerID="ece60ccd8c6bb99fb191b6af8f73697bc478351f1554e1348cc6b2150d08f7a1" Oct 30 20:30:17.037546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.037489 4959 generic.go:356] "Generic (PLEG): container finished" podID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerID="57e0bac9948b1b2cb564085927bed0d7dbf66722352a00d8767465bc29f3e14c" exitCode=60 Oct 30 20:30:17.037546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.037534 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerDied","Data":"57e0bac9948b1b2cb564085927bed0d7dbf66722352a00d8767465bc29f3e14c"} Oct 30 20:30:17.038013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.037951 4959 scope.go:122] "RemoveContainer" containerID="57e0bac9948b1b2cb564085927bed0d7dbf66722352a00d8767465bc29f3e14c" Oct 30 20:30:17.038295 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:17.038271 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp_odh-ai-gateway-infra(150b05b0-19bd-4138-bf01-c5f4e103257e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" Oct 30 20:30:17.039999 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.039966 4959 generic.go:356] "Generic (PLEG): container finished" podID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerID="09c865ff775df9b397a5ced13a25ce83dfbe05c3ac659a186398980de377380e" exitCode=60 Oct 30 20:30:17.040132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.040048 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerDied","Data":"09c865ff775df9b397a5ced13a25ce83dfbe05c3ac659a186398980de377380e"} Oct 30 20:30:17.040428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.040408 4959 scope.go:122] "RemoveContainer" containerID="09c865ff775df9b397a5ced13a25ce83dfbe05c3ac659a186398980de377380e" Oct 30 20:30:17.040728 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:17.040707 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7_odh-ai-gateway-infra(e4cbe136-f3d8-431c-a9a6-9c970fb258c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" Oct 30 20:30:17.047738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.047718 4959 scope.go:122] "RemoveContainer" containerID="0f03f0a0e8fcf6eeb0f49adb82c624a52e2c5537ea0bc96ea99306dde8df6b20" Oct 30 20:30:17.061867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.061845 4959 scope.go:122] "RemoveContainer" containerID="ba682c9438e8771e992dc92e6987b85ae17b750660301f5e101cfab3f382589d" Oct 30 20:30:17.094323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.094301 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:17.173520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.172038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:17.178718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.178689 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:17.217390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.217359 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 20:30:17.224223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.224191 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-284bf6-57d488bf67-z99cx"] Oct 30 20:30:17.315275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.315180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:17.925966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:17.925937 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:18.015155 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.015121 4959 scope.go:122] "RemoveContainer" containerID="ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2" Oct 30 20:30:18.066052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.066017 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a8512e-1144-4d81-a115-51de804273cc" path="/var/lib/kubelet/pods/d3a8512e-1144-4d81-a115-51de804273cc/volumes" Oct 30 20:30:18.308299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.308276 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:18.310982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.310960 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:18.409393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409359 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\") pod \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " Oct 30 20:30:18.409613 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409442 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\") pod \"150b05b0-19bd-4138-bf01-c5f4e103257e\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " Oct 30 20:30:18.409707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409684 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\" (UniqueName: \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\") pod \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\" (UID: \"e4cbe136-f3d8-431c-a9a6-9c970fb258c0\") " Oct 30 20:30:18.409775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409743 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\" (UniqueName: \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\") pod \"150b05b0-19bd-4138-bf01-c5f4e103257e\" (UID: \"150b05b0-19bd-4138-bf01-c5f4e103257e\") " Oct 30 20:30:18.409836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409804 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca" pod "e4cbe136-f3d8-431c-a9a6-9c970fb258c0" (UID: "e4cbe136-f3d8-431c-a9a6-9c970fb258c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:18.409836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.409820 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca" pod "150b05b0-19bd-4138-bf01-c5f4e103257e" (UID: "150b05b0-19bd-4138-bf01-c5f4e103257e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:18.410257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.410231 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:18.410489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.410261 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150b05b0-19bd-4138-bf01-c5f4e103257e-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:18.412328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.412299 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds" pod "e4cbe136-f3d8-431c-a9a6-9c970fb258c0" (UID: "e4cbe136-f3d8-431c-a9a6-9c970fb258c0"). InnerVolumeSpecName "kube-api-access-6dwds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:18.412656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.412635 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj" pod "150b05b0-19bd-4138-bf01-c5f4e103257e" (UID: "150b05b0-19bd-4138-bf01-c5f4e103257e"). InnerVolumeSpecName "kube-api-access-h4plj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:18.512342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.512249 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4plj\" (UniqueName: \"kubernetes.io/projected/150b05b0-19bd-4138-bf01-c5f4e103257e-kube-api-access-h4plj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:18.512342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.512290 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwds\" (UniqueName: \"kubernetes.io/projected/e4cbe136-f3d8-431c-a9a6-9c970fb258c0-kube-api-access-6dwds\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:18.989206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:18.989181 4959 scope.go:122] "RemoveContainer" containerID="a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75" Oct 30 20:30:19.081481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.081442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" event={"ID":"150b05b0-19bd-4138-bf01-c5f4e103257e","Type":"ContainerDied","Data":"d57720fe5b2228c37ff9c7f8aa872782c7d79a44529233d91fb00db8429a3897"} Oct 30 20:30:19.081948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.081543 4959 scope.go:122] "RemoveContainer" containerID="57e0bac9948b1b2cb564085927bed0d7dbf66722352a00d8767465bc29f3e14c" Oct 30 20:30:19.081948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.081701 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp" Oct 30 20:30:19.087267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.087243 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" event={"ID":"e4cbe136-f3d8-431c-a9a6-9c970fb258c0","Type":"ContainerDied","Data":"78cb95a677cf210cf410c397fc72c2a9905f4be5c33966d5369e44ee1124c0a7"} Oct 30 20:30:19.087384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.087342 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7" Oct 30 20:30:19.093557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.093531 4959 generic.go:356] "Generic (PLEG): container finished" podID="95a0b3fe-00dd-4085-a295-822662873fc6" containerID="fa463c3a77b8fbe4b0c84886813177b27c48864f834e407aa9fead54fa7a68ee" exitCode=60 Oct 30 20:30:19.093671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.093600 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerDied","Data":"fa463c3a77b8fbe4b0c84886813177b27c48864f834e407aa9fead54fa7a68ee"} Oct 30 20:30:19.094054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.094035 4959 scope.go:122] "RemoveContainer" containerID="fa463c3a77b8fbe4b0c84886813177b27c48864f834e407aa9fead54fa7a68ee" Oct 30 20:30:19.094366 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:19.094343 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5_odh-ai-gateway-infra(95a0b3fe-00dd-4085-a295-822662873fc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" Oct 30 20:30:19.095625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.095609 4959 scope.go:122] "RemoveContainer" containerID="09c865ff775df9b397a5ced13a25ce83dfbe05c3ac659a186398980de377380e" Oct 30 20:30:19.104669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.104647 4959 scope.go:122] "RemoveContainer" containerID="ebd86f551ee450a4874304c3f538f160241ecbacb56d5e984c34798dbe9d86a2" Oct 30 20:30:19.153165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.153140 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:19.156197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.156175 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:19.193577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.193553 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-27a4d3-29889870-ndctp"] Oct 30 20:30:19.288782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.288751 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:19.332519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.328611 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:19.335309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.334341 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5c2e32-29889870-h88q7"] Oct 30 20:30:19.999305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.999239 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" path="/var/lib/kubelet/pods/150b05b0-19bd-4138-bf01-c5f4e103257e/volumes" Oct 30 20:30:19.999810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:19.999789 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" path="/var/lib/kubelet/pods/e4cbe136-f3d8-431c-a9a6-9c970fb258c0/volumes" Oct 30 20:30:20.101429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.101357 4959 generic.go:356] "Generic (PLEG): container finished" podID="88143ed0-d601-4e9e-8086-65378962508b" containerID="ce3b955167eac9ff346b8e821cef345290d7f1d33e1ad1c71e76de6166ff9fce" exitCode=60 Oct 30 20:30:20.101911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.101439 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerDied","Data":"ce3b955167eac9ff346b8e821cef345290d7f1d33e1ad1c71e76de6166ff9fce"} Oct 30 20:30:20.101911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.101528 4959 scope.go:122] "RemoveContainer" containerID="a9e1353a6151191a8741ffc91b93e3555a42b1b4485bd2284d7b18edc4e85e75" Oct 30 20:30:20.102038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.101950 4959 scope.go:122] "RemoveContainer" containerID="ce3b955167eac9ff346b8e821cef345290d7f1d33e1ad1c71e76de6166ff9fce" Oct 30 20:30:20.102304 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:20.102280 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4_odh-ai-gateway-infra(88143ed0-d601-4e9e-8086-65378962508b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" podUID="88143ed0-d601-4e9e-8086-65378962508b" Oct 30 20:30:20.290738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.290704 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:20.443125 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.443100 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:20.544944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.544913 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\") pod \"95a0b3fe-00dd-4085-a295-822662873fc6\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " Oct 30 20:30:20.545139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.544967 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\" (UniqueName: \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\") pod \"95a0b3fe-00dd-4085-a295-822662873fc6\" (UID: \"95a0b3fe-00dd-4085-a295-822662873fc6\") " Oct 30 20:30:20.545303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.545279 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca" pod "95a0b3fe-00dd-4085-a295-822662873fc6" (UID: "95a0b3fe-00dd-4085-a295-822662873fc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:20.545539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.545486 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a0b3fe-00dd-4085-a295-822662873fc6-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:20.547301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.547279 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v" pod "95a0b3fe-00dd-4085-a295-822662873fc6" (UID: "95a0b3fe-00dd-4085-a295-822662873fc6"). InnerVolumeSpecName "kube-api-access-bhj5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:20.646420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.646390 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhj5v\" (UniqueName: \"kubernetes.io/projected/95a0b3fe-00dd-4085-a295-822662873fc6-kube-api-access-bhj5v\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:20.758051 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.757961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:20.866916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:20.866886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:21.114572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.114440 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" event={"ID":"95a0b3fe-00dd-4085-a295-822662873fc6","Type":"ContainerDied","Data":"4174bc021ce57470b625cfa450672ec884fbb3f42e1bcc6a39a8c6a16792791c"} Oct 30 20:30:21.114572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.114446 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5" Oct 30 20:30:21.114572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.114531 4959 scope.go:122] "RemoveContainer" containerID="fa463c3a77b8fbe4b0c84886813177b27c48864f834e407aa9fead54fa7a68ee" Oct 30 20:30:21.122416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.122391 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:21.151266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.151238 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:21.169003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.168533 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:21.174817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.174794 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-109cfe-29889870-79rp5"] Oct 30 20:30:21.359059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.359033 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:21.454011 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.453976 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\" (UniqueName: \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\") pod \"88143ed0-d601-4e9e-8086-65378962508b\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " Oct 30 20:30:21.454219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.454052 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\") pod \"88143ed0-d601-4e9e-8086-65378962508b\" (UID: \"88143ed0-d601-4e9e-8086-65378962508b\") " Oct 30 20:30:21.454420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.454397 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca" pod "88143ed0-d601-4e9e-8086-65378962508b" (UID: "88143ed0-d601-4e9e-8086-65378962508b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:21.454563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.454543 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88143ed0-d601-4e9e-8086-65378962508b-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:21.456361 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.456339 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5" pod "88143ed0-d601-4e9e-8086-65378962508b" (UID: "88143ed0-d601-4e9e-8086-65378962508b"). InnerVolumeSpecName "kube-api-access-4wvs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:21.555998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.555964 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvs5\" (UniqueName: \"kubernetes.io/projected/88143ed0-d601-4e9e-8086-65378962508b-kube-api-access-4wvs5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:21.583151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.583117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:21.997200 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:21.997164 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" path="/var/lib/kubelet/pods/95a0b3fe-00dd-4085-a295-822662873fc6/volumes" Oct 30 20:30:22.123468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:22.123436 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" Oct 30 20:30:22.123969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:22.123434 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4" event={"ID":"88143ed0-d601-4e9e-8086-65378962508b","Type":"ContainerDied","Data":"1f63a4ef5ec60703daa1e9eaa33f2f519a6b07569b444508c63d3b8f9903a124"} Oct 30 20:30:22.123969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:22.123570 4959 scope.go:122] "RemoveContainer" containerID="ce3b955167eac9ff346b8e821cef345290d7f1d33e1ad1c71e76de6166ff9fce" Oct 30 20:30:22.171358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:22.171333 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:22.177418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:22.177391 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-01f7d3-29889870-kg9w4"] Oct 30 20:30:23.588903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:23.588873 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:23.997966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:23.997926 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88143ed0-d601-4e9e-8086-65378962508b" path="/var/lib/kubelet/pods/88143ed0-d601-4e9e-8086-65378962508b/volumes" Oct 30 20:30:25.401049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:25.401021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:27.277396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:27.277362 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:28.398713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:28.398685 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:30:28.548292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:28.548257 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:28.669021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:28.668934 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:29.443736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:29.443702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:29.754105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:29.754013 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:29.882075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:29.882031 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:31.240677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:31.240647 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:31.742470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:31.742435 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:31.882689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:31.882657 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.029517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.029408 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.166336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.166302 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 20:30:32.166914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.166877 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" podUID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" containerName="maas-api" containerID="cri-o://8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c" gracePeriod=30 Oct 30 20:30:32.336378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.336346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.454481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.454453 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 20:30:32.531857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.531803 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.607889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.607817 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\" (UniqueName: \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\") pod \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " Oct 30 20:30:32.608050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.607893 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\") pod \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " Oct 30 20:30:32.608050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.607913 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\") pod \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " Oct 30 20:30:32.608050 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.607947 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\") pod \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\" (UID: \"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8\") " Oct 30 20:30:32.608399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.608375 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca" pod "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" (UID: "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:32.610281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.610256 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls" pod "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" (UID: "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:32.610683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.610665 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs" pod "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" (UID: "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:32.610772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.610694 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl" pod "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" (UID: "17d94575-6bf2-45a5-a7f2-f72dd9a14ae8"). InnerVolumeSpecName "kube-api-access-pmhpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:32.626087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.625638 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 20:30:32.670542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.670485 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.709446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.709418 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.709446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.709445 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.709688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.709456 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.709688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.709466 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhpl\" (UniqueName: \"kubernetes.io/projected/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8-kube-api-access-pmhpl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:32.790645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.790608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:32.915006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:32.914969 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.073310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.073261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.189714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.189618 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.194913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.194886 4959 generic.go:356] "Generic (PLEG): container finished" podID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" containerID="8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c" exitCode=0 Oct 30 20:30:33.195074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.194930 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" Oct 30 20:30:33.195074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.194973 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" event={"ID":"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8","Type":"ContainerDied","Data":"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c"} Oct 30 20:30:33.195074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.195032 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh" event={"ID":"17d94575-6bf2-45a5-a7f2-f72dd9a14ae8","Type":"ContainerDied","Data":"7d51b44b5e0d44fb9550e2df39a024a1a37948d5ff0c37aa70d47e75a09f0295"} Oct 30 20:30:33.195074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.195062 4959 scope.go:122] "RemoveContainer" containerID="8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c" Oct 30 20:30:33.207379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.207359 4959 scope.go:122] "RemoveContainer" containerID="8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c" Oct 30 20:30:33.207686 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:33.207664 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c\": container with ID starting with 8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c not found: ID does not exist" containerID="8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c" Oct 30 20:30:33.207783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.207693 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c"} err="failed to get container status \"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c\": rpc error: code = NotFound desc = could not find container \"8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c\": container with ID starting with 8ce7fe229f2436fb342165215f22c053e230bac676a8a85f9ca896491bf6c58c not found: ID does not exist" Oct 30 20:30:33.293143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.293111 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 20:30:33.326041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.326002 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-284bf6-554dc47cbc-mvzxh"] Oct 30 20:30:33.373012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.372976 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.399089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.399055 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" containerID="cri-o://a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7" gracePeriod=25 Oct 30 20:30:33.753587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.753514 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.870858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.870820 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.997890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.997843 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:33.998638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:33.998611 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" path="/var/lib/kubelet/pods/17d94575-6bf2-45a5-a7f2-f72dd9a14ae8/volumes" Oct 30 20:30:34.345867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:34.345838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:36.053096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:36.053056 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:36.358663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:36.358581 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:36.682885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:36.682842 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:37.633518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:37.631556 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" containerID="cri-o://bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461" gracePeriod=25 Oct 30 20:30:38.347087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.342629 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:30:38.347087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.342866 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" podUID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" containerName="maas-api" containerID="cri-o://86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158" gracePeriod=30 Oct 30 20:30:38.411689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.411646 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:38.651196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.649707 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:30:38.749239 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.749200 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") pod \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " Oct 30 20:30:38.749448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.749325 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\" (UniqueName: \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\") pod \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " Oct 30 20:30:38.749627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.749450 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\") pod \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " Oct 30 20:30:38.749627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.749545 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\") pod \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\" (UID: \"a8df0448-d3cb-4d7c-8315-bbfeb8206530\") " Oct 30 20:30:38.750160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.750117 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca" pod "a8df0448-d3cb-4d7c-8315-bbfeb8206530" (UID: "a8df0448-d3cb-4d7c-8315-bbfeb8206530"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:38.752695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.752665 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls" pod "a8df0448-d3cb-4d7c-8315-bbfeb8206530" (UID: "a8df0448-d3cb-4d7c-8315-bbfeb8206530"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:38.752825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.752766 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs" pod "a8df0448-d3cb-4d7c-8315-bbfeb8206530" (UID: "a8df0448-d3cb-4d7c-8315-bbfeb8206530"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:38.754087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.754062 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr" pod "a8df0448-d3cb-4d7c-8315-bbfeb8206530" (UID: "a8df0448-d3cb-4d7c-8315-bbfeb8206530"). InnerVolumeSpecName "kube-api-access-9gmzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:38.865630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.851210 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8df0448-d3cb-4d7c-8315-bbfeb8206530-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:38.865630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.851254 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:38.865630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.851269 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8df0448-d3cb-4d7c-8315-bbfeb8206530-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:38.865630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:38.851281 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmzr\" (UniqueName: \"kubernetes.io/projected/a8df0448-d3cb-4d7c-8315-bbfeb8206530-kube-api-access-9gmzr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:39.134897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.134860 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.248358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.248321 4959 generic.go:356] "Generic (PLEG): container finished" podID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" containerID="86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158" exitCode=0 Oct 30 20:30:39.248584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.248382 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" Oct 30 20:30:39.248584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.248397 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" event={"ID":"a8df0448-d3cb-4d7c-8315-bbfeb8206530","Type":"ContainerDied","Data":"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158"} Oct 30 20:30:39.248584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.248449 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps" event={"ID":"a8df0448-d3cb-4d7c-8315-bbfeb8206530","Type":"ContainerDied","Data":"1bf6b266893de4e15216655a455386cd644eb532548d4375c1de7ff822c1b45d"} Oct 30 20:30:39.248584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.248473 4959 scope.go:122] "RemoveContainer" containerID="86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158" Oct 30 20:30:39.260406 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.260383 4959 scope.go:122] "RemoveContainer" containerID="86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158" Oct 30 20:30:39.262862 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:39.262835 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158\": container with ID starting with 86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158 not found: ID does not exist" containerID="86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158" Oct 30 20:30:39.262969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.262872 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158"} err="failed to get container status \"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158\": rpc error: code = NotFound desc = could not find container \"86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158\": container with ID starting with 86dfdcbb8dd332a03d749a095b73fe6d2e98473d4b18b13e626ff59373225158 not found: ID does not exist" Oct 30 20:30:39.270386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.270358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.323561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.323524 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:30:39.340725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.340685 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5314c2-7476586768-v9xps"] Oct 30 20:30:39.418728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.418654 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.830930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:39.830835 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:40.000162 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:40.000127 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" path="/var/lib/kubelet/pods/a8df0448-d3cb-4d7c-8315-bbfeb8206530/volumes" Oct 30 20:30:41.374601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:41.374151 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:41.520443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:41.520398 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:41.649952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:41.649911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:43.669207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:43.669166 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:43.823777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:43.823738 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:43.943449 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:43.943364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.229206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:44.229122 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.894168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:44.894136 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:45.014485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.014448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:45.197931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.197856 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:30:45.198186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.198154 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" containerID="cri-o://1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2" gracePeriod=30 Oct 30 20:30:45.220419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.220392 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:45.342276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.342200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:45.554001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.553917 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:45.733677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:45.733643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:46.065880 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:46.065844 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:30:46.681971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:46.681930 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:47.367617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:47.367578 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:47.455935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:47.455897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:47.558078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:47.558049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:47.738273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:47.738239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:47.897366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:47.897339 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.012531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:48.012180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.807322 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:48.807287 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:30:48.807818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:48.807361 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:30:49.500648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:49.500612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.191779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.191739 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.317832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.317802 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.482274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.482246 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:30:50.602657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602623 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.602833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602713 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.602833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602755 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.602833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602811 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602838 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602950 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602980 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.602988 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.603236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603025 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603126 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.603236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603138 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\") pod \"93bfdb23-7483-4676-ac5b-36a7cde42d55\" (UID: \"93bfdb23-7483-4676-ac5b-36a7cde42d55\") " Oct 30 20:30:50.603382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603312 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.603382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603339 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:50.603482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603430 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.603729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603705 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bfdb23-7483-4676-ac5b-36a7cde42d55-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.603962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603732 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.603962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603749 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.603962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603758 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.603962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.603767 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.605292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.605263 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:30:50.605479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.605460 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "kube-api-access-lk27l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:50.605607 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.605589 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:30:50.605646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.605622 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token" pod "93bfdb23-7483-4676-ac5b-36a7cde42d55" (UID: "93bfdb23-7483-4676-ac5b-36a7cde42d55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:50.704985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.704897 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.704985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.704928 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.704985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.704940 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk27l\" (UniqueName: \"kubernetes.io/projected/93bfdb23-7483-4676-ac5b-36a7cde42d55-kube-api-access-lk27l\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.704985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.704954 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bfdb23-7483-4676-ac5b-36a7cde42d55-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:50.883714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:50.883673 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:51.315663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.315624 4959 generic.go:356] "Generic (PLEG): container finished" podID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerID="1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2" exitCode=0 Oct 30 20:30:51.316177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.315686 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" Oct 30 20:30:51.316177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.315704 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerDied","Data":"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2"} Oct 30 20:30:51.316177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.315759 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss" event={"ID":"93bfdb23-7483-4676-ac5b-36a7cde42d55","Type":"ContainerDied","Data":"8c4fd918bd658dcf95c635ddf1a500153950f8253381bb5b3838439dcb1c9f3f"} Oct 30 20:30:51.316177 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.315784 4959 scope.go:122] "RemoveContainer" containerID="1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2" Oct 30 20:30:51.327083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.326916 4959 scope.go:122] "RemoveContainer" containerID="01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68" Oct 30 20:30:51.336724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.336708 4959 scope.go:122] "RemoveContainer" containerID="1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2" Oct 30 20:30:51.336926 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:51.336910 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2\": container with ID starting with 1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2 not found: ID does not exist" containerID="1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2" Oct 30 20:30:51.336986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.336934 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2"} err="failed to get container status \"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2\": rpc error: code = NotFound desc = could not find container \"1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2\": container with ID starting with 1f2278fde7f7d9468a3c216374efc4b2887eb14d281f9218db1dd32574ed83c2 not found: ID does not exist" Oct 30 20:30:51.336986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.336949 4959 scope.go:122] "RemoveContainer" containerID="01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68" Oct 30 20:30:51.337178 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:51.337156 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68\": container with ID starting with 01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68 not found: ID does not exist" containerID="01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68" Oct 30 20:30:51.337235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.337184 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68"} err="failed to get container status \"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68\": rpc error: code = NotFound desc = could not find container \"01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68\": container with ID starting with 01490fea56de48520d60f902b277a66879892ae62d070c6846ffc8238aec8e68 not found: ID does not exist" Oct 30 20:30:51.383703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.383677 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:30:51.394790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.394767 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-77ac5a-openshift-default-58d4d8b5df-2w5ss"] Oct 30 20:30:51.996374 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:51.996339 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" path="/var/lib/kubelet/pods/93bfdb23-7483-4676-ac5b-36a7cde42d55/volumes" Oct 30 20:30:52.396849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:52.396817 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:53.537665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.537633 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:53.682289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.682264 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:30:53.836353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.836324 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\") pod \"0bea1573-2af5-4655-8184-c008a3f4659b\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " Oct 30 20:30:53.836563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.836368 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\" (UniqueName: \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\") pod \"0bea1573-2af5-4655-8184-c008a3f4659b\" (UID: \"0bea1573-2af5-4655-8184-c008a3f4659b\") " Oct 30 20:30:53.836761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.836732 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume" pod "0bea1573-2af5-4655-8184-c008a3f4659b" (UID: "0bea1573-2af5-4655-8184-c008a3f4659b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:53.836894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.836879 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bea1573-2af5-4655-8184-c008a3f4659b-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:53.838729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.838705 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll" pod "0bea1573-2af5-4655-8184-c008a3f4659b" (UID: "0bea1573-2af5-4655-8184-c008a3f4659b"). InnerVolumeSpecName "kube-api-access-7hwll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:53.937614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:53.937546 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwll\" (UniqueName: \"kubernetes.io/projected/0bea1573-2af5-4655-8184-c008a3f4659b-kube-api-access-7hwll\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:54.196347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.196270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:54.335251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.335214 4959 generic.go:356] "Generic (PLEG): container finished" podID="0bea1573-2af5-4655-8184-c008a3f4659b" containerID="a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7" exitCode=0 Oct 30 20:30:54.335458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.335276 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" event={"ID":"0bea1573-2af5-4655-8184-c008a3f4659b","Type":"ContainerDied","Data":"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7"} Oct 30 20:30:54.335458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.335329 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" event={"ID":"0bea1573-2af5-4655-8184-c008a3f4659b","Type":"ContainerDied","Data":"f888c3eebeccdbfaaad969fbfebd02df73cfe1ed01c0ae9136f46272a81c70c9"} Oct 30 20:30:54.335458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.335330 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9" Oct 30 20:30:54.335458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.335351 4959 scope.go:122] "RemoveContainer" containerID="a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7" Oct 30 20:30:54.344963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.344943 4959 scope.go:122] "RemoveContainer" containerID="a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7" Oct 30 20:30:54.345215 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:54.345195 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7\": container with ID starting with a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7 not found: ID does not exist" containerID="a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7" Oct 30 20:30:54.345291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.345222 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7"} err="failed to get container status \"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7\": rpc error: code = NotFound desc = could not find container \"a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7\": container with ID starting with a4cc86cd9a10d8c90d16961d3d438288f092c2deeda7b7436d6e22838f0557f7 not found: ID does not exist" Oct 30 20:30:54.384310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.384286 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:30:54.395398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:54.395376 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5314c2-7ddd74487b-7nkt9"] Oct 30 20:30:55.934548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:55.934511 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:30:55.942099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:55.942075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:55.965209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:55.965057 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:30:55.996580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:55.996554 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" path="/var/lib/kubelet/pods/0bea1573-2af5-4655-8184-c008a3f4659b/volumes" Oct 30 20:30:56.092447 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:56.092420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:56.224838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:56.224755 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.366943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.366905 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:30:57.525910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.525872 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.634077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.634041 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.917856 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.917831 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.927793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.927770 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 20:30:57.988822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.988792 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 20:30:57.989129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:57.989099 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" containerID="cri-o://9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d" gracePeriod=30 Oct 30 20:30:58.095810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.095779 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\") pod \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " Oct 30 20:30:58.096000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.095868 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\" (UniqueName: \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\") pod \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\" (UID: \"e5c9b1b7-18fe-4476-b31a-8405cd8491b8\") " Oct 30 20:30:58.096256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.096232 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume" pod "e5c9b1b7-18fe-4476-b31a-8405cd8491b8" (UID: "e5c9b1b7-18fe-4476-b31a-8405cd8491b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:58.098254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.098199 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n" pod "e5c9b1b7-18fe-4476-b31a-8405cd8491b8" (UID: "e5c9b1b7-18fe-4476-b31a-8405cd8491b8"). InnerVolumeSpecName "kube-api-access-bqp4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:58.176742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.176659 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.197189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.197164 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:58.197189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.197188 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqp4n\" (UniqueName: \"kubernetes.io/projected/e5c9b1b7-18fe-4476-b31a-8405cd8491b8-kube-api-access-bqp4n\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:30:58.360082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.360051 4959 generic.go:356] "Generic (PLEG): container finished" podID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerID="bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461" exitCode=0 Oct 30 20:30:58.360264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.360098 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" Oct 30 20:30:58.360264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.360116 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" event={"ID":"e5c9b1b7-18fe-4476-b31a-8405cd8491b8","Type":"ContainerDied","Data":"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461"} Oct 30 20:30:58.360264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.360154 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2" event={"ID":"e5c9b1b7-18fe-4476-b31a-8405cd8491b8","Type":"ContainerDied","Data":"9630f99b8fc4ba335df38a5b0c1d79379a847e6aae69902a6102f458019483ae"} Oct 30 20:30:58.360264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.360173 4959 scope.go:122] "RemoveContainer" containerID="bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461" Oct 30 20:30:58.374029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.374006 4959 scope.go:122] "RemoveContainer" containerID="bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461" Oct 30 20:30:58.374381 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:30:58.374313 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461\": container with ID starting with bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461 not found: ID does not exist" containerID="bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461" Oct 30 20:30:58.374381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.374353 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461"} err="failed to get container status \"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461\": rpc error: code = NotFound desc = could not find container \"bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461\": container with ID starting with bb661e15487c9dfd2eef4f85532a5d4b9e73a6590182cdce4b4d04d642336461 not found: ID does not exist" Oct 30 20:30:58.424089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.424061 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 20:30:58.433740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:58.433676 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5c2e32-6f89bcbcbc-2whg2"] Oct 30 20:30:59.001256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.001212 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:59.119709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.119676 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:59.850186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.850151 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:30:59.850600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.850382 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" podUID="6799a8d5-19a3-4186-9433-99cfb35db04f" containerName="maas-api" containerID="cri-o://266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935" gracePeriod=30 Oct 30 20:30:59.941763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.941730 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:59.996417 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:30:59.996373 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" path="/var/lib/kubelet/pods/e5c9b1b7-18fe-4476-b31a-8405cd8491b8/volumes" Oct 30 20:31:00.152738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.152704 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:31:00.244300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244267 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\") pod \"6799a8d5-19a3-4186-9433-99cfb35db04f\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " Oct 30 20:31:00.244300 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244301 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\") pod \"6799a8d5-19a3-4186-9433-99cfb35db04f\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " Oct 30 20:31:00.244556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244433 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\") pod \"6799a8d5-19a3-4186-9433-99cfb35db04f\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " Oct 30 20:31:00.244614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244574 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\" (UniqueName: \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\") pod \"6799a8d5-19a3-4186-9433-99cfb35db04f\" (UID: \"6799a8d5-19a3-4186-9433-99cfb35db04f\") " Oct 30 20:31:00.244670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244650 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca" pod "6799a8d5-19a3-4186-9433-99cfb35db04f" (UID: "6799a8d5-19a3-4186-9433-99cfb35db04f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:00.244937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.244915 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6799a8d5-19a3-4186-9433-99cfb35db04f-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:00.246726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.246697 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44" pod "6799a8d5-19a3-4186-9433-99cfb35db04f" (UID: "6799a8d5-19a3-4186-9433-99cfb35db04f"). InnerVolumeSpecName "kube-api-access-t6l44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:00.246833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.246767 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs" pod "6799a8d5-19a3-4186-9433-99cfb35db04f" (UID: "6799a8d5-19a3-4186-9433-99cfb35db04f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:00.246833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.246785 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls" pod "6799a8d5-19a3-4186-9433-99cfb35db04f" (UID: "6799a8d5-19a3-4186-9433-99cfb35db04f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:00.346134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.346094 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:00.346134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.346131 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6799a8d5-19a3-4186-9433-99cfb35db04f-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:00.346411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.346147 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6l44\" (UniqueName: \"kubernetes.io/projected/6799a8d5-19a3-4186-9433-99cfb35db04f-kube-api-access-t6l44\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:00.373802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.373720 4959 generic.go:356] "Generic (PLEG): container finished" podID="6799a8d5-19a3-4186-9433-99cfb35db04f" containerID="266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935" exitCode=0 Oct 30 20:31:00.373802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.373760 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" Oct 30 20:31:00.374014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.373798 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" event={"ID":"6799a8d5-19a3-4186-9433-99cfb35db04f","Type":"ContainerDied","Data":"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935"} Oct 30 20:31:00.374014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.373843 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r" event={"ID":"6799a8d5-19a3-4186-9433-99cfb35db04f","Type":"ContainerDied","Data":"868ab939d6c9b38c0814b002f6bbcf093fa30111d2baff0d897e98ca0605694d"} Oct 30 20:31:00.374014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.373865 4959 scope.go:122] "RemoveContainer" containerID="266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935" Oct 30 20:31:00.385043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.385023 4959 scope.go:122] "RemoveContainer" containerID="266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935" Oct 30 20:31:00.385340 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:00.385316 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935\": container with ID starting with 266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935 not found: ID does not exist" containerID="266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935" Oct 30 20:31:00.385448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.385346 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935"} err="failed to get container status \"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935\": rpc error: code = NotFound desc = could not find container \"266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935\": container with ID starting with 266c57ab16a969b2ea603f946e080d157df89289de3c7fa6353dec35dcd44935 not found: ID does not exist" Oct 30 20:31:00.429380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.429354 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:31:00.435927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.435905 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5c875d-5984bb4497-dwt4r"] Oct 30 20:31:00.578486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.578450 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.723924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.723883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.846733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.846704 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.934815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.934775 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" containerID="cri-o://2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb" gracePeriod=25 Oct 30 20:31:00.965627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:00.965594 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" containerID="cri-o://7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db" gracePeriod=25 Oct 30 20:31:01.182698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:01.182665 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:01.309406 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:01.309369 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:01.868408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:01.868370 4959 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:31:01.868626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:01.868421 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:31:01.995737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:01.995702 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6799a8d5-19a3-4186-9433-99cfb35db04f" path="/var/lib/kubelet/pods/6799a8d5-19a3-4186-9433-99cfb35db04f/volumes" Oct 30 20:31:02.919891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:02.919859 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:03.257010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.256979 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 20:31:03.378866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.378837 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.378929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.378984 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379058 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379014 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379076 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379197 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379231 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379279 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379311 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\") pod \"de2bc6bf-0003-4e75-aa20-f934692d168c\" (UID: \"de2bc6bf-0003-4e75-aa20-f934692d168c\") " Oct 30 20:31:03.379535 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379367 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:03.379640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379618 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:03.379799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379781 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.379878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379804 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.379919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379869 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:03.379919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379869 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:03.379986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.379962 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:03.381792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.381766 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:03.381792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.381781 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:03.382034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.382013 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "kube-api-access-f2n84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:03.382076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.382032 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token" pod "de2bc6bf-0003-4e75-aa20-f934692d168c" (UID: "de2bc6bf-0003-4e75-aa20-f934692d168c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:03.393299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.393274 4959 generic.go:356] "Generic (PLEG): container finished" podID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerID="9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d" exitCode=0 Oct 30 20:31:03.393394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.393320 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" Oct 30 20:31:03.393394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.393359 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerDied","Data":"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d"} Oct 30 20:31:03.393469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.393414 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv" event={"ID":"de2bc6bf-0003-4e75-aa20-f934692d168c","Type":"ContainerDied","Data":"61aa7b866af615b7239dca413ab18a276b69f8859e18ce41c74158963d101a2b"} Oct 30 20:31:03.393469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.393443 4959 scope.go:122] "RemoveContainer" containerID="9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d" Oct 30 20:31:03.406708 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.406690 4959 scope.go:122] "RemoveContainer" containerID="b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f" Oct 30 20:31:03.415826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.415802 4959 scope.go:122] "RemoveContainer" containerID="9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d" Oct 30 20:31:03.416110 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:03.416092 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d\": container with ID starting with 9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d not found: ID does not exist" containerID="9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d" Oct 30 20:31:03.416161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.416119 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d"} err="failed to get container status \"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d\": rpc error: code = NotFound desc = could not find container \"9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d\": container with ID starting with 9f1b86329063082b9bedc85117d5af65f18a471b55c3d81c9f276089bc68825d not found: ID does not exist" Oct 30 20:31:03.416161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.416138 4959 scope.go:122] "RemoveContainer" containerID="b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f" Oct 30 20:31:03.416381 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:03.416364 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f\": container with ID starting with b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f not found: ID does not exist" containerID="b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f" Oct 30 20:31:03.416443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.416391 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f"} err="failed to get container status \"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f\": rpc error: code = NotFound desc = could not find container \"b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f\": container with ID starting with b67573a6f8744bc18d1f6d89e992ae4c838f6927d2b88dd7c36aa01f75c7f73f not found: ID does not exist" Oct 30 20:31:03.445109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.445000 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 20:31:03.454721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.454690 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5c2e32-openshift-default-5798584d47-99hsv"] Oct 30 20:31:03.480399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480371 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480398 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2bc6bf-0003-4e75-aa20-f934692d168c-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480408 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480416 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480424 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480431 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.480612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.480440 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2n84\" (UniqueName: \"kubernetes.io/projected/de2bc6bf-0003-4e75-aa20-f934692d168c-kube-api-access-f2n84\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:03.996726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:03.996690 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" path="/var/lib/kubelet/pods/de2bc6bf-0003-4e75-aa20-f934692d168c/volumes" Oct 30 20:31:04.496352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:04.496302 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:05.036088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.036054 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:05.153422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.153389 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:05.267378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.267344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:05.599045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.598981 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:05.599045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.599023 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:05.662930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:05.662892 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:07.067927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:07.067881 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:07.873863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:07.873825 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:07.978530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:07.977863 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:08.180087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.180052 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:31:08.180593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.180266 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" containerID="cri-o://cd61ddedc7f8ca9000c06ce54d108d0ca38de5738f34d869a434294d0efd1fce" gracePeriod=30 Oct 30 20:31:08.209997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.209960 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:08.675839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.675804 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 20:31:08.683947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.683912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:08.714309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.714275 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 20:31:08.800185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.800130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:08.921217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:08.921184 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:10.807951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:10.807915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.270847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.270806 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.280112 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.280059 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 20:31:12.280343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.280320 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" podUID="9f51f024-9772-4774-ac32-a994be32bc88" containerName="maas-api" containerID="cri-o://63789771be16c1820433b662009bc835091a40c183d87a387a7efb56a6761701" gracePeriod=30 Oct 30 20:31:12.380174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.380132 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.444042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.444012 4959 generic.go:356] "Generic (PLEG): container finished" podID="9f51f024-9772-4774-ac32-a994be32bc88" containerID="63789771be16c1820433b662009bc835091a40c183d87a387a7efb56a6761701" exitCode=0 Oct 30 20:31:12.444196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.444090 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" event={"ID":"9f51f024-9772-4774-ac32-a994be32bc88","Type":"ContainerDied","Data":"63789771be16c1820433b662009bc835091a40c183d87a387a7efb56a6761701"} Oct 30 20:31:12.569715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.569682 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 20:31:12.576420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.576386 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:31:12.576710 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.576665 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" containerID="cri-o://7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579" gracePeriod=30 Oct 30 20:31:12.620031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.619999 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.684626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.684593 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\") pod \"9f51f024-9772-4774-ac32-a994be32bc88\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " Oct 30 20:31:12.684814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.684651 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") pod \"9f51f024-9772-4774-ac32-a994be32bc88\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " Oct 30 20:31:12.684814 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.684789 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\") pod \"9f51f024-9772-4774-ac32-a994be32bc88\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " Oct 30 20:31:12.684908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.684830 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\" (UniqueName: \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\") pod \"9f51f024-9772-4774-ac32-a994be32bc88\" (UID: \"9f51f024-9772-4774-ac32-a994be32bc88\") " Oct 30 20:31:12.685136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.685111 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca" pod "9f51f024-9772-4774-ac32-a994be32bc88" (UID: "9f51f024-9772-4774-ac32-a994be32bc88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:12.685309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.685284 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f51f024-9772-4774-ac32-a994be32bc88-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:12.687080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.687058 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79" pod "9f51f024-9772-4774-ac32-a994be32bc88" (UID: "9f51f024-9772-4774-ac32-a994be32bc88"). InnerVolumeSpecName "kube-api-access-l8q79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:12.687362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.687335 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls" pod "9f51f024-9772-4774-ac32-a994be32bc88" (UID: "9f51f024-9772-4774-ac32-a994be32bc88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:12.687362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.687346 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs" pod "9f51f024-9772-4774-ac32-a994be32bc88" (UID: "9f51f024-9772-4774-ac32-a994be32bc88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:12.786646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.786551 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8q79\" (UniqueName: \"kubernetes.io/projected/9f51f024-9772-4774-ac32-a994be32bc88-kube-api-access-l8q79\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:12.786646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.786585 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:12.786646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.786597 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f51f024-9772-4774-ac32-a994be32bc88-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:12.915876 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:12.915835 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.018965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.018926 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.131170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.131086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.450537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.450478 4959 generic.go:356] "Generic (PLEG): container finished" podID="d591a945-0928-4519-ad6a-52c378a44d51" containerID="cd61ddedc7f8ca9000c06ce54d108d0ca38de5738f34d869a434294d0efd1fce" exitCode=0 Oct 30 20:31:13.450935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.450551 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" event={"ID":"d591a945-0928-4519-ad6a-52c378a44d51","Type":"ContainerDied","Data":"cd61ddedc7f8ca9000c06ce54d108d0ca38de5738f34d869a434294d0efd1fce"} Oct 30 20:31:13.450935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.450605 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" event={"ID":"d591a945-0928-4519-ad6a-52c378a44d51","Type":"ContainerDied","Data":"dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c"} Oct 30 20:31:13.450935 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.450630 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd2ed958e7801eeaa032c8d175adc1192f19e6c651fbf7dc78e16527c27e036c" Oct 30 20:31:13.451948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.451924 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" Oct 30 20:31:13.452079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.451945 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r" event={"ID":"9f51f024-9772-4774-ac32-a994be32bc88","Type":"ContainerDied","Data":"d1ac5c430cf0a892d70fc6ae3e80783dff018072a1ba6542bb1947d3ac41fd73"} Oct 30 20:31:13.452079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.452001 4959 scope.go:122] "RemoveContainer" containerID="63789771be16c1820433b662009bc835091a40c183d87a387a7efb56a6761701" Oct 30 20:31:13.452961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.452940 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:31:13.501785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.501750 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 20:31:13.513924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.513892 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-27a4d3-bf685f85b-k266r"] Oct 30 20:31:13.558035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.558001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.594711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594679 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.594916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594722 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.594916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594783 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.594916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594806 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.594916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594838 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.594916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594896 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.594965 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595069 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\") pod \"d591a945-0928-4519-ad6a-52c378a44d51\" (UID: \"d591a945-0928-4519-ad6a-52c378a44d51\") " Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595103 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595119 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.595215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595137 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.595629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595285 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:13.595629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595544 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.595758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595739 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.595817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595762 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.595817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595777 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.595817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595790 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.595817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.595804 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d591a945-0928-4519-ad6a-52c378a44d51-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.597565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.597540 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:13.597692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.597589 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "kube-api-access-xslfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:13.597894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.597869 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:13.597970 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.597896 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy" pod "d591a945-0928-4519-ad6a-52c378a44d51" (UID: "d591a945-0928-4519-ad6a-52c378a44d51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:13.654479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.654446 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 20:31:13.676524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.676467 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" containerID="cri-o://d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20" gracePeriod=25 Oct 30 20:31:13.696631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.696550 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d591a945-0928-4519-ad6a-52c378a44d51-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.696631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.696581 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslfl\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-kube-api-access-xslfl\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.696631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.696591 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d591a945-0928-4519-ad6a-52c378a44d51-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.696631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.696602 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d591a945-0928-4519-ad6a-52c378a44d51-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:13.715047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.714985 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" containerID="cri-o://5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880" gracePeriod=25 Oct 30 20:31:13.765917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.765877 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.997197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:13.997116 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f51f024-9772-4774-ac32-a994be32bc88" path="/var/lib/kubelet/pods/9f51f024-9772-4774-ac32-a994be32bc88/volumes" Oct 30 20:31:14.458758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:14.458732 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj" Oct 30 20:31:14.521004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:14.520969 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:31:14.532045 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:14.532016 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5c875d-openshift-default-6cb7465cfc-cn7sj"] Oct 30 20:31:14.996390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:14.996362 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:15.598208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:15.598164 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:15.598208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:15.598196 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:15.998318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:15.998284 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d591a945-0928-4519-ad6a-52c378a44d51" path="/var/lib/kubelet/pods/d591a945-0928-4519-ad6a-52c378a44d51/volumes" Oct 30 20:31:16.065161 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:16.065122 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:31:16.798309 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:16.798274 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:17.838983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.838955 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:31:17.940038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.939999 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940043 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940091 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940148 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940169 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940195 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940225 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940321 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940271 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940326 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\") pod \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\" (UID: \"e9362571-8f8f-4bde-ae48-95ceec8c60fd\") " Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940410 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940425 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940648 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940660 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:17.940695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940662 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:17.940968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940795 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:17.940968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940815 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:17.940968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940829 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:17.940968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940842 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:17.940968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.940856 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:17.942555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.942527 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:17.942707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.942688 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "kube-api-access-9z5vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:17.942828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.942813 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:17.943089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:17.943075 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token" pod "e9362571-8f8f-4bde-ae48-95ceec8c60fd" (UID: "e9362571-8f8f-4bde-ae48-95ceec8c60fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:18.041958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.041924 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:18.041958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.041954 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:18.042147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.041968 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9362571-8f8f-4bde-ae48-95ceec8c60fd-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:18.042147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.041980 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5vz\" (UniqueName: \"kubernetes.io/projected/e9362571-8f8f-4bde-ae48-95ceec8c60fd-kube-api-access-9z5vz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:18.072736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.072705 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 20:31:18.077886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.077861 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 20:31:18.098568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.098540 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 20:31:18.207313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.207224 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:18.480767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.480662 4959 generic.go:356] "Generic (PLEG): container finished" podID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerID="7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579" exitCode=0 Oct 30 20:31:18.480767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.480713 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" Oct 30 20:31:18.480767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.480746 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" event={"ID":"e9362571-8f8f-4bde-ae48-95ceec8c60fd","Type":"ContainerDied","Data":"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579"} Oct 30 20:31:18.481012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.480810 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79" event={"ID":"e9362571-8f8f-4bde-ae48-95ceec8c60fd","Type":"ContainerDied","Data":"7c3ab10cc7e1806a17c4b33efef87a1ab7fc15db42201c167723caf8f5edbdd2"} Oct 30 20:31:18.481012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.480833 4959 scope.go:122] "RemoveContainer" containerID="7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579" Oct 30 20:31:18.490404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.490385 4959 scope.go:122] "RemoveContainer" containerID="7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579" Oct 30 20:31:18.490769 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:18.490745 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579\": container with ID starting with 7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579 not found: ID does not exist" containerID="7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579" Oct 30 20:31:18.490848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.490778 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579"} err="failed to get container status \"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579\": rpc error: code = NotFound desc = could not find container \"7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579\": container with ID starting with 7cee5201d2738164028db71c06e1a7787875776ec06e0e868088cf9763fcf579 not found: ID does not exist" Oct 30 20:31:18.512190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.512161 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:18.528114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.528091 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:31:18.539143 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:18.539113 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9d2720-openshift-default-77c6cdfc87-dbt79"] Oct 30 20:31:19.026812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:19.026728 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 20:31:19.167893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:19.167858 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:19.275157 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:19.275125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:19.997104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:19.997070 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" path="/var/lib/kubelet/pods/e9362571-8f8f-4bde-ae48-95ceec8c60fd/volumes" Oct 30 20:31:20.249998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:20.249903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:20.735077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:20.735042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:20.844735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:20.844704 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:21.090721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.090694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:21.237000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.236933 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:31:21.278605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.278569 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:21.279049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.278572 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:21.292561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.292536 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:31:21.391020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.390981 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\") pod \"0dd16a23-dd26-4487-805c-598bd2cee8ff\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " Oct 30 20:31:21.391204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.391055 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\" (UniqueName: \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\") pod \"0dd16a23-dd26-4487-805c-598bd2cee8ff\" (UID: \"0dd16a23-dd26-4487-805c-598bd2cee8ff\") " Oct 30 20:31:21.391204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.391151 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\") pod \"9202623e-d88f-4f90-b72d-92cd8716eddf\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " Oct 30 20:31:21.391204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.391183 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\" (UniqueName: \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\") pod \"9202623e-d88f-4f90-b72d-92cd8716eddf\" (UID: \"9202623e-d88f-4f90-b72d-92cd8716eddf\") " Oct 30 20:31:21.391576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.391541 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume" pod "0dd16a23-dd26-4487-805c-598bd2cee8ff" (UID: "0dd16a23-dd26-4487-805c-598bd2cee8ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:21.391742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.391585 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume" pod "9202623e-d88f-4f90-b72d-92cd8716eddf" (UID: "9202623e-d88f-4f90-b72d-92cd8716eddf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:21.393560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.393537 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw" pod "0dd16a23-dd26-4487-805c-598bd2cee8ff" (UID: "0dd16a23-dd26-4487-805c-598bd2cee8ff"). InnerVolumeSpecName "kube-api-access-vgzzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:21.393683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.393631 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp" pod "9202623e-d88f-4f90-b72d-92cd8716eddf" (UID: "9202623e-d88f-4f90-b72d-92cd8716eddf"). InnerVolumeSpecName "kube-api-access-fkxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:21.492934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.492897 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd16a23-dd26-4487-805c-598bd2cee8ff-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:21.492934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.492928 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgzzw\" (UniqueName: \"kubernetes.io/projected/0dd16a23-dd26-4487-805c-598bd2cee8ff-kube-api-access-vgzzw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:21.492934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.492937 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9202623e-d88f-4f90-b72d-92cd8716eddf-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:21.492934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.492945 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxfp\" (UniqueName: \"kubernetes.io/projected/9202623e-d88f-4f90-b72d-92cd8716eddf-kube-api-access-fkxfp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:21.500941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.500917 4959 generic.go:356] "Generic (PLEG): container finished" podID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerID="2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb" exitCode=0 Oct 30 20:31:21.501088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.500968 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" Oct 30 20:31:21.501088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.501003 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" event={"ID":"0dd16a23-dd26-4487-805c-598bd2cee8ff","Type":"ContainerDied","Data":"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb"} Oct 30 20:31:21.501088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.501059 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh" event={"ID":"0dd16a23-dd26-4487-805c-598bd2cee8ff","Type":"ContainerDied","Data":"1f58bb011885d1af5172ed7946a06446f5dad2cedcbb30ed929a60c5f3975a34"} Oct 30 20:31:21.501088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.501077 4959 scope.go:122] "RemoveContainer" containerID="2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb" Oct 30 20:31:21.502355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.502334 4959 generic.go:356] "Generic (PLEG): container finished" podID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerID="7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db" exitCode=0 Oct 30 20:31:21.502450 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.502365 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" event={"ID":"9202623e-d88f-4f90-b72d-92cd8716eddf","Type":"ContainerDied","Data":"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db"} Oct 30 20:31:21.502450 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.502377 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" Oct 30 20:31:21.502450 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.502418 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5" event={"ID":"9202623e-d88f-4f90-b72d-92cd8716eddf","Type":"ContainerDied","Data":"e79ff40361985e2fa920d3b88e6cd6d16177e17b6d24dbfc5584fb4b290d3f49"} Oct 30 20:31:21.512552 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.512398 4959 scope.go:122] "RemoveContainer" containerID="2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb" Oct 30 20:31:21.512862 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:21.512813 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb\": container with ID starting with 2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb not found: ID does not exist" containerID="2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb" Oct 30 20:31:21.512862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.512852 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb"} err="failed to get container status \"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb\": rpc error: code = NotFound desc = could not find container \"2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb\": container with ID starting with 2896d423c1c4fc5681ebbace48a4f4942dfd4adb0b860dbbd98ebdeffcc897bb not found: ID does not exist" Oct 30 20:31:21.513007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.512874 4959 scope.go:122] "RemoveContainer" containerID="7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db" Oct 30 20:31:21.521898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.521881 4959 scope.go:122] "RemoveContainer" containerID="7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db" Oct 30 20:31:21.522202 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:21.522158 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db\": container with ID starting with 7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db not found: ID does not exist" containerID="7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db" Oct 30 20:31:21.522202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.522187 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db"} err="failed to get container status \"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db\": rpc error: code = NotFound desc = could not find container \"7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db\": container with ID starting with 7510bfdad9cb6b773d5514ba8d1ebb23a5e0101f6bde60f281a9fe8724d232db not found: ID does not exist" Oct 30 20:31:21.548182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.548158 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:31:21.551001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.550980 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9d2720-988659b6c-pd5z5"] Oct 30 20:31:21.552813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.552792 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:21.581320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.581285 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:31:21.585770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.585750 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-9d2720-6d5b886f64-dgszh"] Oct 30 20:31:21.818833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.818748 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw"] Oct 30 20:31:21.819404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819381 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9f51f024-9772-4774-ac32-a994be32bc88" containerName="maas-api" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819406 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f51f024-9772-4774-ac32-a994be32bc88" containerName="maas-api" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819428 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819436 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819450 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819458 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819472 4959 cpu_manager.go:414] "containerMap: removing container" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819479 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819488 4959 cpu_manager.go:414] "containerMap: removing container" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819513 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819528 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819535 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.819539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819546 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819553 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819569 4959 cpu_manager.go:414] "containerMap: removing container" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819577 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819592 4959 cpu_manager.go:414] "containerMap: removing container" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819599 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819610 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6799a8d5-19a3-4186-9433-99cfb35db04f" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819616 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6799a8d5-19a3-4186-9433-99cfb35db04f" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819628 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819635 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819652 4959 cpu_manager.go:414] "containerMap: removing container" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819659 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819667 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819674 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819686 4959 cpu_manager.go:414] "containerMap: removing container" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819693 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819702 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819708 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819726 4959 cpu_manager.go:414] "containerMap: removing container" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819732 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819743 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819750 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819763 4959 cpu_manager.go:414] "containerMap: removing container" podUID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819771 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819785 4959 cpu_manager.go:414] "containerMap: removing container" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819791 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819805 4959 cpu_manager.go:414] "containerMap: removing container" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819813 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819826 4959 cpu_manager.go:414] "containerMap: removing container" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819834 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819848 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819856 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819870 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819877 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819888 4959 cpu_manager.go:414] "containerMap: removing container" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819894 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819905 4959 cpu_manager.go:414] "containerMap: removing container" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819911 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819923 4959 cpu_manager.go:414] "containerMap: removing container" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819931 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819946 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.819952 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820086 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a8512e-1144-4d81-a115-51de804273cc" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820098 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820111 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" containerName="payload-pre-processing" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820120 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bfdb23-7483-4676-ac5b-36a7cde42d55" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820130 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9362571-8f8f-4bde-ae48-95ceec8c60fd" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820141 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820150 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820158 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d94575-6bf2-45a5-a7f2-f72dd9a14ae8" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820168 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820176 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820185 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820193 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8df0448-d3cb-4d7c-8315-bbfeb8206530" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820203 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="928c98ce-c4d5-417d-a715-e92d39072d88" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820211 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b62495-d391-4c86-b5c6-0f5989657f27" containerName="main" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820220 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820229 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d591a945-0928-4519-ad6a-52c378a44d51" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820239 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820247 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820254 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f51f024-9772-4774-ac32-a994be32bc88" containerName="maas-api" Oct 30 20:31:21.820221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820263 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c9b1b7-18fe-4476-b31a-8405cd8491b8" containerName="payload-pre-processing" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820272 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" containerName="payload-processing" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820282 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e13789-931c-44a4-bbeb-60d874d39502" containerName="main" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820291 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820301 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bea1573-2af5-4655-8184-c008a3f4659b" containerName="payload-processing" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820311 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:21.823617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.820320 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6799a8d5-19a3-4186-9433-99cfb35db04f" containerName="maas-api" Oct 30 20:31:21.825254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.825234 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:21.840806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.840695 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw"] Oct 30 20:31:21.877328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.877287 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:31:21.877328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.877311 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:31:21.996724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.996692 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd16a23-dd26-4487-805c-598bd2cee8ff" path="/var/lib/kubelet/pods/0dd16a23-dd26-4487-805c-598bd2cee8ff/volumes" Oct 30 20:31:21.997097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.997083 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9202623e-d88f-4f90-b72d-92cd8716eddf" path="/var/lib/kubelet/pods/9202623e-d88f-4f90-b72d-92cd8716eddf/volumes" Oct 30 20:31:21.997728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.997710 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68sm8\" (UniqueName: \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:21.997809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.997754 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:21.997894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:21.997868 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.099105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.099015 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68sm8\" (UniqueName: \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.099105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.099077 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.099105 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.099107 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.099879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.099858 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.101699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.101681 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.117234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.117211 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68sm8\" (UniqueName: \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\") pod \"maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.138902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.138877 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:22.361834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.361752 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:22.445859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.445824 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw"] Oct 30 20:31:22.451049 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:31:22.451020 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33039e57_f4c9_4e41_ade3_7b7d44ec8583.slice/crio-3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236 WatchSource:0}: Error finding container 3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236: Status 404 returned error can't find the container with id 3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236 Oct 30 20:31:22.509817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:22.509786 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" event={"ID":"33039e57-f4c9-4e41-ade3-7b7d44ec8583","Type":"ContainerStarted","Data":"3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236"} Oct 30 20:31:23.518077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:23.518041 4959 generic.go:356] "Generic (PLEG): container finished" podID="33039e57-f4c9-4e41-ade3-7b7d44ec8583" containerID="33791713d5ebf7eab61a0a1f23725041e1962643d77c410ad67878e30d9552ab" exitCode=0 Oct 30 20:31:23.518543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:23.518136 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" event={"ID":"33039e57-f4c9-4e41-ade3-7b7d44ec8583","Type":"ContainerDied","Data":"33791713d5ebf7eab61a0a1f23725041e1962643d77c410ad67878e30d9552ab"} Oct 30 20:31:23.602940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:23.602854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:23.716619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:23.716584 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:24.028073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.028033 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" podUID="3cfef63e-8821-451f-9403-fc3b7356f408" containerName="payload-processing" containerID="cri-o://9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b" gracePeriod=25 Oct 30 20:31:24.279394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.279340 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 20:31:24.318569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.318544 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\") pod \"3cfef63e-8821-451f-9403-fc3b7356f408\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " Oct 30 20:31:24.318701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.318661 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\" (UniqueName: \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\") pod \"3cfef63e-8821-451f-9403-fc3b7356f408\" (UID: \"3cfef63e-8821-451f-9403-fc3b7356f408\") " Oct 30 20:31:24.319034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.318894 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume" pod "3cfef63e-8821-451f-9403-fc3b7356f408" (UID: "3cfef63e-8821-451f-9403-fc3b7356f408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:24.319186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.319034 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cfef63e-8821-451f-9403-fc3b7356f408-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.322523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.322466 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t" pod "3cfef63e-8821-451f-9403-fc3b7356f408" (UID: "3cfef63e-8821-451f-9403-fc3b7356f408"). InnerVolumeSpecName "kube-api-access-rld5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:24.405558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.405525 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:24.420113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.420089 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rld5t\" (UniqueName: \"kubernetes.io/projected/3cfef63e-8821-451f-9403-fc3b7356f408-kube-api-access-rld5t\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.525563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.525528 4959 generic.go:356] "Generic (PLEG): container finished" podID="3cfef63e-8821-451f-9403-fc3b7356f408" containerID="9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b" exitCode=0 Oct 30 20:31:24.526031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.525613 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" event={"ID":"3cfef63e-8821-451f-9403-fc3b7356f408","Type":"ContainerDied","Data":"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b"} Oct 30 20:31:24.526031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.525667 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" event={"ID":"3cfef63e-8821-451f-9403-fc3b7356f408","Type":"ContainerDied","Data":"28d4048413eb93722aabc164debc2a10d05d6f42f01e23eac9261697828953c5"} Oct 30 20:31:24.526031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.525689 4959 scope.go:122] "RemoveContainer" containerID="9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b" Oct 30 20:31:24.526031 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.525737 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs" Oct 30 20:31:24.537431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.537411 4959 scope.go:122] "RemoveContainer" containerID="9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b" Oct 30 20:31:24.537739 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:24.537721 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b\": container with ID starting with 9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b not found: ID does not exist" containerID="9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b" Oct 30 20:31:24.537791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.537748 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b"} err="failed to get container status \"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b\": rpc error: code = NotFound desc = could not find container \"9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b\": container with ID starting with 9c4dc296d13a589f9915c28034ea69ba7f3c0ddfe50500a989a6192e40c2936b not found: ID does not exist" Oct 30 20:31:24.573581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.573549 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 20:31:24.578570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.578482 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-5db436-6c54f888bb-89gcs"] Oct 30 20:31:24.665371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.665347 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:24.723380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.723353 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\" (UniqueName: \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\") pod \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " Oct 30 20:31:24.723555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.723467 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\") pod \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " Oct 30 20:31:24.723555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.723508 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\") pod \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\" (UID: \"33039e57-f4c9-4e41-ade3-7b7d44ec8583\") " Oct 30 20:31:24.723900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.723876 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca" pod "33039e57-f4c9-4e41-ade3-7b7d44ec8583" (UID: "33039e57-f4c9-4e41-ade3-7b7d44ec8583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:24.725634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.725608 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8" pod "33039e57-f4c9-4e41-ade3-7b7d44ec8583" (UID: "33039e57-f4c9-4e41-ade3-7b7d44ec8583"). InnerVolumeSpecName "kube-api-access-68sm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:24.725714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.725695 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca" pod "33039e57-f4c9-4e41-ade3-7b7d44ec8583" (UID: "33039e57-f4c9-4e41-ade3-7b7d44ec8583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:24.824654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.824571 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.824654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.824605 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33039e57-f4c9-4e41-ade3-7b7d44ec8583-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:24.824654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:24.824617 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68sm8\" (UniqueName: \"kubernetes.io/projected/33039e57-f4c9-4e41-ade3-7b7d44ec8583-kube-api-access-68sm8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:25.533161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:25.533132 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" Oct 30 20:31:25.533577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:25.533162 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw" event={"ID":"33039e57-f4c9-4e41-ade3-7b7d44ec8583","Type":"ContainerDied","Data":"3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236"} Oct 30 20:31:25.533577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:25.533212 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f660ad42d93a28e839aa2b9fd0b4d2b6ab96077856db8856fe8840f18126236" Oct 30 20:31:25.998166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:25.998131 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfef63e-8821-451f-9403-fc3b7356f408" path="/var/lib/kubelet/pods/3cfef63e-8821-451f-9403-fc3b7356f408/volumes" Oct 30 20:31:26.415870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:26.415838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:26.519743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:26.519710 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 20:31:26.530850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:26.530819 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 20:31:26.534573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:26.534368 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 20:31:26.613023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:26.612987 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:28.178027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.177990 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:28.676989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.676946 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" containerID="cri-o://71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96" gracePeriod=45 Oct 30 20:31:28.954180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.954110 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 20:31:28.972487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972453 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.972667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972526 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.972667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972601 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.972667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972662 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.972855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972693 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.972855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972704 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:28.972855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972738 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\" (UniqueName: \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\") pod \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\" (UID: \"17d1c6a4-47e9-4ecb-8597-63602b53e3b9\") " Oct 30 20:31:28.973037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.972852 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:28.973037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.973007 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:28.973189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.973170 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:28.973234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.973195 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:28.973234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.973209 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:28.975142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.975117 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:28.975436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.975416 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:28.975557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:28.975539 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz" pod "17d1c6a4-47e9-4ecb-8597-63602b53e3b9" (UID: "17d1c6a4-47e9-4ecb-8597-63602b53e3b9"). InnerVolumeSpecName "kube-api-access-7zwmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:29.074171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.074137 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.074171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.074171 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.074382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.074189 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwmz\" (UniqueName: \"kubernetes.io/projected/17d1c6a4-47e9-4ecb-8597-63602b53e3b9-kube-api-access-7zwmz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:29.563861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.563825 4959 generic.go:356] "Generic (PLEG): container finished" podID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerID="71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96" exitCode=0 Oct 30 20:31:29.564347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.563890 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" Oct 30 20:31:29.564608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.563914 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" event={"ID":"17d1c6a4-47e9-4ecb-8597-63602b53e3b9","Type":"ContainerDied","Data":"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96"} Oct 30 20:31:29.564696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.564638 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m" event={"ID":"17d1c6a4-47e9-4ecb-8597-63602b53e3b9","Type":"ContainerDied","Data":"ead1d8c87e57375257046e63ad4a03489d13f8b04288b3181a0823c0bf0b2190"} Oct 30 20:31:29.564696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.564671 4959 scope.go:122] "RemoveContainer" containerID="71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96" Oct 30 20:31:29.575823 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.575806 4959 scope.go:122] "RemoveContainer" containerID="71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96" Oct 30 20:31:29.577035 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:29.577011 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96\": container with ID starting with 71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96 not found: ID does not exist" containerID="71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96" Oct 30 20:31:29.577144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.577042 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96"} err="failed to get container status \"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96\": rpc error: code = NotFound desc = could not find container \"71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96\": container with ID starting with 71707d71aef07da764b4d1fc394cb5c3ddccf1a89440c2596f4d014a64a76a96 not found: ID does not exist" Oct 30 20:31:29.611854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.611820 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 20:31:29.622968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.622927 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5db436/distinct-w0-5db436-kserve-768cc9c8dc-ljv6m"] Oct 30 20:31:29.998289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:29.998236 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" path="/var/lib/kubelet/pods/17d1c6a4-47e9-4ecb-8597-63602b53e3b9/volumes" Oct 30 20:31:30.016972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:30.016941 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:31.277506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:31.277471 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:31.277921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:31.277527 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:31.692419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:31.692389 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:32.001452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:32.001361 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 20:31:32.033064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:32.033028 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 20:31:32.143149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:32.143121 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:32.247158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:32.247128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:33.094378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.094343 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" containerID="cri-o://fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c" gracePeriod=45 Oct 30 20:31:33.098089 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.098057 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" containerID="cri-o://7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f" gracePeriod=45 Oct 30 20:31:33.118660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.118628 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" containerID="cri-o://b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667" gracePeriod=45 Oct 30 20:31:33.381451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.381418 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 20:31:33.387201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.387174 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 20:31:33.410579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.410547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420476 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420553 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\" (UniqueName: \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420622 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420653 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\" (UniqueName: \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420681 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420736 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420776 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420829 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420869 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\") pod \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\" (UID: \"4d5d4f89-f494-42e7-aa33-f37b64e8048c\") " Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420898 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420985 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.421010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421013 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.421587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.420917 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421099 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\" (UniqueName: \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421153 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\") pod \"da4db64c-00b4-417d-9bb3-d83c466c79be\" (UID: \"da4db64c-00b4-417d-9bb3-d83c466c79be\") " Oct 30 20:31:33.421587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421578 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.421750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421640 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.421750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421662 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.422186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.421965 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.425676 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.425801 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "kube-api-access-t55p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.425948 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.428891 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.422120 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.429532 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.438521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.433980 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.445298 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.445233 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d" pod "da4db64c-00b4-417d-9bb3-d83c466c79be" (UID: "da4db64c-00b4-417d-9bb3-d83c466c79be"). InnerVolumeSpecName "kube-api-access-pq48d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:33.445593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.445552 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm" pod "4d5d4f89-f494-42e7-aa33-f37b64e8048c" (UID: "4d5d4f89-f494-42e7-aa33-f37b64e8048c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.522727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.522692 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.522894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.522751 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.522960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.522906 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.523018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.522987 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.523018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.522991 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\" (UniqueName: \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.523134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523042 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\" (UniqueName: \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.523134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523094 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\") pod \"857c2303-7886-445b-8fe9-d1326b68e879\" (UID: \"857c2303-7886-445b-8fe9-d1326b68e879\") " Oct 30 20:31:33.523268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523242 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.523790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523759 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523834 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523853 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523866 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d4f89-f494-42e7-aa33-f37b64e8048c-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523879 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d4f89-f494-42e7-aa33-f37b64e8048c-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523891 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.523912 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523903 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.524127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523916 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq48d\" (UniqueName: \"kubernetes.io/projected/da4db64c-00b4-417d-9bb3-d83c466c79be-kube-api-access-pq48d\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.524127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523930 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4db64c-00b4-417d-9bb3-d83c466c79be-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.524127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523942 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4db64c-00b4-417d-9bb3-d83c466c79be-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.524127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523955 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55p6\" (UniqueName: \"kubernetes.io/projected/4d5d4f89-f494-42e7-aa33-f37b64e8048c-kube-api-access-t55p6\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.524127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.523967 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.525812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.525782 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:33.525956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.525900 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:33.526238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.526219 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr" pod "857c2303-7886-445b-8fe9-d1326b68e879" (UID: "857c2303-7886-445b-8fe9-d1326b68e879"). InnerVolumeSpecName "kube-api-access-w72wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:33.596653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.596618 4959 generic.go:356] "Generic (PLEG): container finished" podID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerID="fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c" exitCode=0 Oct 30 20:31:33.596841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.596680 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" Oct 30 20:31:33.596841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.596705 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" event={"ID":"4d5d4f89-f494-42e7-aa33-f37b64e8048c","Type":"ContainerDied","Data":"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c"} Oct 30 20:31:33.596841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.596745 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" event={"ID":"4d5d4f89-f494-42e7-aa33-f37b64e8048c","Type":"ContainerDied","Data":"78d5b2f5c564f5d6e7599c1bcdeb5c55637b37a93b19754aa8354d24c5e709a6"} Oct 30 20:31:33.596841 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.596762 4959 scope.go:122] "RemoveContainer" containerID="fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c" Oct 30 20:31:33.598644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.598618 4959 generic.go:356] "Generic (PLEG): container finished" podID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerID="7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f" exitCode=0 Oct 30 20:31:33.598773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.598733 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" Oct 30 20:31:33.598848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.598763 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" event={"ID":"da4db64c-00b4-417d-9bb3-d83c466c79be","Type":"ContainerDied","Data":"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f"} Oct 30 20:31:33.598848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.598799 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" event={"ID":"da4db64c-00b4-417d-9bb3-d83c466c79be","Type":"ContainerDied","Data":"ff728ce803b01f45bff90ae4193860a80e1ea43d1bb34eb38911547b1063dc8b"} Oct 30 20:31:33.600259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.600223 4959 generic.go:356] "Generic (PLEG): container finished" podID="857c2303-7886-445b-8fe9-d1326b68e879" containerID="b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667" exitCode=0 Oct 30 20:31:33.600393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.600281 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" Oct 30 20:31:33.600393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.600309 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" event={"ID":"857c2303-7886-445b-8fe9-d1326b68e879","Type":"ContainerDied","Data":"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667"} Oct 30 20:31:33.600393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.600350 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng" event={"ID":"857c2303-7886-445b-8fe9-d1326b68e879","Type":"ContainerDied","Data":"b99cd97bef5922fa60492a6dc9916e2e82ff623872bdc72826287d73a721675c"} Oct 30 20:31:33.607764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.607744 4959 scope.go:122] "RemoveContainer" containerID="fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c" Oct 30 20:31:33.608059 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:33.608039 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c\": container with ID starting with fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c not found: ID does not exist" containerID="fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c" Oct 30 20:31:33.608141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.608073 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c"} err="failed to get container status \"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c\": rpc error: code = NotFound desc = could not find container \"fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c\": container with ID starting with fe2ae82637ee8a319d7769f6638c962aa2e31ceaf14b2ebbfaeb64ca933c962c not found: ID does not exist" Oct 30 20:31:33.608141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.608098 4959 scope.go:122] "RemoveContainer" containerID="7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f" Oct 30 20:31:33.617758 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.617743 4959 scope.go:122] "RemoveContainer" containerID="7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f" Oct 30 20:31:33.617999 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:33.617979 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f\": container with ID starting with 7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f not found: ID does not exist" containerID="7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f" Oct 30 20:31:33.618048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.618005 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f"} err="failed to get container status \"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f\": rpc error: code = NotFound desc = could not find container \"7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f\": container with ID starting with 7067a24cdf9cf172de417a50d210694df2a61dec58532785db4767d1849ba25f not found: ID does not exist" Oct 30 20:31:33.618048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.618022 4959 scope.go:122] "RemoveContainer" containerID="b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667" Oct 30 20:31:33.629335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.629203 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72wr\" (UniqueName: \"kubernetes.io/projected/857c2303-7886-445b-8fe9-d1326b68e879-kube-api-access-w72wr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.629335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.629235 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/857c2303-7886-445b-8fe9-d1326b68e879-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.629335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.629293 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.629335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.629308 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/857c2303-7886-445b-8fe9-d1326b68e879-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:33.640206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.636285 4959 scope.go:122] "RemoveContainer" containerID="b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667" Oct 30 20:31:33.640206 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:33.636711 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667\": container with ID starting with b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667 not found: ID does not exist" containerID="b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667" Oct 30 20:31:33.640206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.636751 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667"} err="failed to get container status \"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667\": rpc error: code = NotFound desc = could not find container \"b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667\": container with ID starting with b7f2db888e4f1f893cd08722ae58d8d594c29bc555caa786fee288cf2924f667 not found: ID does not exist" Oct 30 20:31:33.647726 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.647701 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 20:31:33.652818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.652076 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk"] Oct 30 20:31:33.710431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.710397 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 20:31:33.714282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.714254 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/simulator-w1-109cfe-kserve-56c9fc549b-wpbng"] Oct 30 20:31:33.746716 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.746681 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 20:31:33.752342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.752316 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9"] Oct 30 20:31:33.922397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.922364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:33.994432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.994398 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 20:31:33.996343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.996320 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" path="/var/lib/kubelet/pods/4d5d4f89-f494-42e7-aa33-f37b64e8048c/volumes" Oct 30 20:31:33.996917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.996896 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857c2303-7886-445b-8fe9-d1326b68e879" path="/var/lib/kubelet/pods/857c2303-7886-445b-8fe9-d1326b68e879/volumes" Oct 30 20:31:33.997435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:33.997416 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" path="/var/lib/kubelet/pods/da4db64c-00b4-417d-9bb3-d83c466c79be/volumes" Oct 30 20:31:34.004806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.004784 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 20:31:34.033484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.033454 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\" (UniqueName: \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\") pod \"142619ed-6462-41a5-aa2d-c0faf35bbf48\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " Oct 30 20:31:34.033798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.033584 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\") pod \"142619ed-6462-41a5-aa2d-c0faf35bbf48\" (UID: \"142619ed-6462-41a5-aa2d-c0faf35bbf48\") " Oct 30 20:31:34.034097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.034054 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume" pod "142619ed-6462-41a5-aa2d-c0faf35bbf48" (UID: "142619ed-6462-41a5-aa2d-c0faf35bbf48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:34.036366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.036341 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq" pod "142619ed-6462-41a5-aa2d-c0faf35bbf48" (UID: "142619ed-6462-41a5-aa2d-c0faf35bbf48"). InnerVolumeSpecName "kube-api-access-m8qjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:34.134806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.134721 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\" (UniqueName: \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\") pod \"baf2369e-1631-445b-ac9a-1eef4b74d87a\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " Oct 30 20:31:34.135248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.134830 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\") pod \"baf2369e-1631-445b-ac9a-1eef4b74d87a\" (UID: \"baf2369e-1631-445b-ac9a-1eef4b74d87a\") " Oct 30 20:31:34.135248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.135166 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142619ed-6462-41a5-aa2d-c0faf35bbf48-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:34.135248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.135172 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume" pod "baf2369e-1631-445b-ac9a-1eef4b74d87a" (UID: "baf2369e-1631-445b-ac9a-1eef4b74d87a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:34.135248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.135186 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qjq\" (UniqueName: \"kubernetes.io/projected/142619ed-6462-41a5-aa2d-c0faf35bbf48-kube-api-access-m8qjq\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:34.137052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.137022 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj" pod "baf2369e-1631-445b-ac9a-1eef4b74d87a" (UID: "baf2369e-1631-445b-ac9a-1eef4b74d87a"). InnerVolumeSpecName "kube-api-access-z4pvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:34.236039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.236002 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baf2369e-1631-445b-ac9a-1eef4b74d87a-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:34.236039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.236033 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pvj\" (UniqueName: \"kubernetes.io/projected/baf2369e-1631-445b-ac9a-1eef4b74d87a-kube-api-access-z4pvj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:34.330886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.330842 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/premium-w1-109cfe-kserve-64b64f4f85-hp6g9" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 20:31:34.363039 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.363000 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-109cfe/distinct-w1-109cfe-kserve-74b69d6f76-qlgmk" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:31:34.419153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.419065 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:34.607598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.607563 4959 generic.go:356] "Generic (PLEG): container finished" podID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerID="5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880" exitCode=0 Oct 30 20:31:34.607784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.607615 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" Oct 30 20:31:34.607784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.607651 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" event={"ID":"baf2369e-1631-445b-ac9a-1eef4b74d87a","Type":"ContainerDied","Data":"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880"} Oct 30 20:31:34.607784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.607708 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk" event={"ID":"baf2369e-1631-445b-ac9a-1eef4b74d87a","Type":"ContainerDied","Data":"793d9e12c08ffe41e090dd615360a3dc7ffbce1c2eb07cd5c524aca9a446e387"} Oct 30 20:31:34.607784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.607736 4959 scope.go:122] "RemoveContainer" containerID="5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880" Oct 30 20:31:34.609142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.609117 4959 generic.go:356] "Generic (PLEG): container finished" podID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerID="d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20" exitCode=0 Oct 30 20:31:34.609210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.609149 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" Oct 30 20:31:34.609271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.609200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" event={"ID":"142619ed-6462-41a5-aa2d-c0faf35bbf48","Type":"ContainerDied","Data":"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20"} Oct 30 20:31:34.609271 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.609254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch" event={"ID":"142619ed-6462-41a5-aa2d-c0faf35bbf48","Type":"ContainerDied","Data":"310a5849141d4b16537aee065d099c15cc796e241fd65e0a002efa0c963b6885"} Oct 30 20:31:34.618884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.618869 4959 scope.go:122] "RemoveContainer" containerID="5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880" Oct 30 20:31:34.619140 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:34.619108 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880\": container with ID starting with 5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880 not found: ID does not exist" containerID="5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880" Oct 30 20:31:34.619212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.619140 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880"} err="failed to get container status \"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880\": rpc error: code = NotFound desc = could not find container \"5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880\": container with ID starting with 5c65c7dc7c9e5b30c01bf71636933fe54a04d06895077a86917ca7b669949880 not found: ID does not exist" Oct 30 20:31:34.619212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.619165 4959 scope.go:122] "RemoveContainer" containerID="d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20" Oct 30 20:31:34.628222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.628204 4959 scope.go:122] "RemoveContainer" containerID="d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20" Oct 30 20:31:34.628457 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:34.628440 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20\": container with ID starting with d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20 not found: ID does not exist" containerID="d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20" Oct 30 20:31:34.628530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.628464 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20"} err="failed to get container status \"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20\": rpc error: code = NotFound desc = could not find container \"d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20\": container with ID starting with d646ef82d06ddc1e9885d1dc6746c1f16aabbc824c7844159922e96e98462f20 not found: ID does not exist" Oct 30 20:31:34.654322 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.654297 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 20:31:34.665317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.665292 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-27a4d3-56dc96d464-644ch"] Oct 30 20:31:34.696741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.696655 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 20:31:34.698451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.698425 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-27a4d3-6c6f7c47b7-mk2nk"] Oct 30 20:31:34.949170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:34.949098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:35.321733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.321656 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99"] Oct 30 20:31:35.322236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322212 4959 cpu_manager.go:414] "containerMap: removing container" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:35.322236 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322235 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2bc6bf-0003-4e75-aa20-f934692d168c" containerName="istio-proxy" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322280 4959 cpu_manager.go:414] "containerMap: removing container" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322296 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322309 4959 cpu_manager.go:414] "containerMap: removing container" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322317 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322330 4959 cpu_manager.go:414] "containerMap: removing container" podUID="33039e57-f4c9-4e41-ade3-7b7d44ec8583" containerName="revoke-keys" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322338 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33039e57-f4c9-4e41-ade3-7b7d44ec8583" containerName="revoke-keys" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322350 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322356 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322379 4959 cpu_manager.go:414] "containerMap: removing container" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322385 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322404 4959 cpu_manager.go:414] "containerMap: removing container" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322411 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322437 4959 cpu_manager.go:414] "containerMap: removing container" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:35.322451 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322444 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322459 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3cfef63e-8821-451f-9403-fc3b7356f408" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322466 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfef63e-8821-451f-9403-fc3b7356f408" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322479 4959 cpu_manager.go:414] "containerMap: removing container" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322486 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322513 4959 cpu_manager.go:414] "containerMap: removing container" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322528 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322633 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="857c2303-7886-445b-8fe9-d1326b68e879" containerName="main" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322647 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="33039e57-f4c9-4e41-ade3-7b7d44ec8583" containerName="revoke-keys" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322658 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="88143ed0-d601-4e9e-8086-65378962508b" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322670 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5d4f89-f494-42e7-aa33-f37b64e8048c" containerName="main" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322682 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a0b3fe-00dd-4085-a295-822662873fc6" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322694 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" containerName="payload-pre-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322705 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d1c6a4-47e9-4ecb-8597-63602b53e3b9" containerName="main" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322717 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfef63e-8821-451f-9403-fc3b7356f408" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322725 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbe136-f3d8-431c-a9a6-9c970fb258c0" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322740 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" containerName="payload-processing" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322748 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="150b05b0-19bd-4138-bf01-c5f4e103257e" containerName="cleanup" Oct 30 20:31:35.323191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.322760 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4db64c-00b4-417d-9bb3-d83c466c79be" containerName="main" Oct 30 20:31:35.329248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.329204 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.344549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.344523 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99"] Oct 30 20:31:35.430066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.430035 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:31:35.430248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.430094 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:31:35.441813 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.441790 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:35.447034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.447008 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt7z\" (UniqueName: \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.447144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.447076 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.447215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.447196 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.547886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.547850 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.548103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.547914 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt7z\" (UniqueName: \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.548103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.548004 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.548600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.548570 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.550560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.550542 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.566062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.566025 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt7z\" (UniqueName: \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\") pod \"maas-api-revoke-sub-4ff98e46-68r99\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.579725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.579656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:35.647000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.646968 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:35.948533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.948482 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99"] Oct 30 20:31:35.953465 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:31:35.953442 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fe0673_9ebd_4de5_a67c_6828c712a279.slice/crio-583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3 WatchSource:0}: Error finding container 583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3: Status 404 returned error can't find the container with id 583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3 Oct 30 20:31:35.996396 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.996368 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142619ed-6462-41a5-aa2d-c0faf35bbf48" path="/var/lib/kubelet/pods/142619ed-6462-41a5-aa2d-c0faf35bbf48/volumes" Oct 30 20:31:35.996956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:35.996941 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf2369e-1631-445b-ac9a-1eef4b74d87a" path="/var/lib/kubelet/pods/baf2369e-1631-445b-ac9a-1eef4b74d87a/volumes" Oct 30 20:31:36.117837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:36.117807 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:36.627297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:36.627209 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" event={"ID":"39fe0673-9ebd-4de5-a67c-6828c712a279","Type":"ContainerStarted","Data":"6b950922d56ecfecdfb75102ca09c16e519c25d15607b4aeacd114569c5a429f"} Oct 30 20:31:36.627297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:36.627272 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" event={"ID":"39fe0673-9ebd-4de5-a67c-6828c712a279","Type":"ContainerStarted","Data":"583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3"} Oct 30 20:31:36.665464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:36.665412 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" podStartSLOduration=1.665399856 podStartE2EDuration="1.665399856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:36.664209289 +0000 UTC m=+3420.793443210" watchObservedRunningTime="2026-10-30 20:31:36.665399856 +0000 UTC m=+3420.794633771" Oct 30 20:31:37.002057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.002009 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" containerID="cri-o://c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad" gracePeriod=25 Oct 30 20:31:37.007243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.007204 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 20:31:37.007482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.007436 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" podUID="527f1372-f22d-450b-bca2-09d79e750070" containerName="maas-api" containerID="cri-o://51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7" gracePeriod=30 Oct 30 20:31:37.033989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.033926 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" containerID="cri-o://496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690" gracePeriod=25 Oct 30 20:31:37.073797 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.073765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:37.256851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.256752 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 20:31:37.367005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.366972 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") pod \"527f1372-f22d-450b-bca2-09d79e750070\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " Oct 30 20:31:37.367005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.367010 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\" (UniqueName: \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\") pod \"527f1372-f22d-450b-bca2-09d79e750070\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " Oct 30 20:31:37.367265 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.367156 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\") pod \"527f1372-f22d-450b-bca2-09d79e750070\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " Oct 30 20:31:37.367265 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.367210 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\") pod \"527f1372-f22d-450b-bca2-09d79e750070\" (UID: \"527f1372-f22d-450b-bca2-09d79e750070\") " Oct 30 20:31:37.367548 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.367520 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca" pod "527f1372-f22d-450b-bca2-09d79e750070" (UID: "527f1372-f22d-450b-bca2-09d79e750070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:37.367652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.367610 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/527f1372-f22d-450b-bca2-09d79e750070-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:37.369247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.369221 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls" pod "527f1372-f22d-450b-bca2-09d79e750070" (UID: "527f1372-f22d-450b-bca2-09d79e750070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:37.369365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.369286 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h" pod "527f1372-f22d-450b-bca2-09d79e750070" (UID: "527f1372-f22d-450b-bca2-09d79e750070"). InnerVolumeSpecName "kube-api-access-s5t2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:37.369365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.369327 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs" pod "527f1372-f22d-450b-bca2-09d79e750070" (UID: "527f1372-f22d-450b-bca2-09d79e750070"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:37.468475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.468442 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:37.468475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.468473 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/527f1372-f22d-450b-bca2-09d79e750070-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:37.468475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.468482 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5t2h\" (UniqueName: \"kubernetes.io/projected/527f1372-f22d-450b-bca2-09d79e750070-kube-api-access-s5t2h\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:37.634085 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.634046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:37.635812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.635787 4959 generic.go:356] "Generic (PLEG): container finished" podID="39fe0673-9ebd-4de5-a67c-6828c712a279" containerID="6b950922d56ecfecdfb75102ca09c16e519c25d15607b4aeacd114569c5a429f" exitCode=0 Oct 30 20:31:37.635954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.635823 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" event={"ID":"39fe0673-9ebd-4de5-a67c-6828c712a279","Type":"ContainerDied","Data":"6b950922d56ecfecdfb75102ca09c16e519c25d15607b4aeacd114569c5a429f"} Oct 30 20:31:37.637218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.637192 4959 generic.go:356] "Generic (PLEG): container finished" podID="527f1372-f22d-450b-bca2-09d79e750070" containerID="51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7" exitCode=0 Oct 30 20:31:37.637348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.637234 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" Oct 30 20:31:37.637348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.637231 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" event={"ID":"527f1372-f22d-450b-bca2-09d79e750070","Type":"ContainerDied","Data":"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7"} Oct 30 20:31:37.637525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.637369 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h" event={"ID":"527f1372-f22d-450b-bca2-09d79e750070","Type":"ContainerDied","Data":"30d2859139f40ed27844bfbd719294ec8df2bad7b51cc781ab53e50877a0f0eb"} Oct 30 20:31:37.637525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.637404 4959 scope.go:122] "RemoveContainer" containerID="51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7" Oct 30 20:31:37.648319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.648301 4959 scope.go:122] "RemoveContainer" containerID="51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7" Oct 30 20:31:37.648653 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:37.648635 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7\": container with ID starting with 51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7 not found: ID does not exist" containerID="51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7" Oct 30 20:31:37.648701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.648661 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7"} err="failed to get container status \"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7\": rpc error: code = NotFound desc = could not find container \"51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7\": container with ID starting with 51384075e1647b6441008605f57d153cec6ac28ea2e04298b9b517bbd13644c7 not found: ID does not exist" Oct 30 20:31:37.710825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.710793 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 20:31:37.713721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.713692 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-109cfe-79d759d8c6-2w86h"] Oct 30 20:31:37.762240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.762208 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:37.995526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:37.995475 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527f1372-f22d-450b-bca2-09d79e750070" path="/var/lib/kubelet/pods/527f1372-f22d-450b-bca2-09d79e750070/volumes" Oct 30 20:31:38.278840 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.278758 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:38.786985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.786419 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:38.880744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.880715 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\" (UniqueName: \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\") pod \"39fe0673-9ebd-4de5-a67c-6828c712a279\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " Oct 30 20:31:38.880969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.880831 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\") pod \"39fe0673-9ebd-4de5-a67c-6828c712a279\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " Oct 30 20:31:38.880969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.880876 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\") pod \"39fe0673-9ebd-4de5-a67c-6828c712a279\" (UID: \"39fe0673-9ebd-4de5-a67c-6828c712a279\") " Oct 30 20:31:38.881299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.881269 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca" pod "39fe0673-9ebd-4de5-a67c-6828c712a279" (UID: "39fe0673-9ebd-4de5-a67c-6828c712a279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:38.883129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.883096 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca" pod "39fe0673-9ebd-4de5-a67c-6828c712a279" (UID: "39fe0673-9ebd-4de5-a67c-6828c712a279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:38.883129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.883099 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z" pod "39fe0673-9ebd-4de5-a67c-6828c712a279" (UID: "39fe0673-9ebd-4de5-a67c-6828c712a279"). InnerVolumeSpecName "kube-api-access-2tt7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:38.981770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.981691 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:38.981770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.981723 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe0673-9ebd-4de5-a67c-6828c712a279-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:38.981770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:38.981734 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tt7z\" (UniqueName: \"kubernetes.io/projected/39fe0673-9ebd-4de5-a67c-6828c712a279-kube-api-access-2tt7z\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:39.649729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:39.649696 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" Oct 30 20:31:39.649898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:39.649717 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99" event={"ID":"39fe0673-9ebd-4de5-a67c-6828c712a279","Type":"ContainerDied","Data":"583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3"} Oct 30 20:31:39.649898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:39.649779 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583e7716f10404f733c32d5b9d74d8f07031a183caeacb02b8a21d63f8e5c2a3" Oct 30 20:31:41.542621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.542550 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" containerID="cri-o://ed86fb8ddecc6706f5ab99bebd27a2f344b95eb3935fcc156c13dee21b7fb59e" gracePeriod=45 Oct 30 20:31:41.556692 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.556529 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" containerID="cri-o://5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c" gracePeriod=45 Oct 30 20:31:41.556891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.556856 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" containerID="cri-o://182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17" gracePeriod=45 Oct 30 20:31:41.665219 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.665113 4959 generic.go:356] "Generic (PLEG): container finished" podID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerID="ed86fb8ddecc6706f5ab99bebd27a2f344b95eb3935fcc156c13dee21b7fb59e" exitCode=0 Oct 30 20:31:41.665519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.665172 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" event={"ID":"aaecb09e-0a11-4b21-b333-2f853774f3b5","Type":"ContainerDied","Data":"ed86fb8ddecc6706f5ab99bebd27a2f344b95eb3935fcc156c13dee21b7fb59e"} Oct 30 20:31:41.833776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.833747 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 20:31:41.890233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.890203 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 20:31:41.892822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.892793 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 20:31:41.914120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914076 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914171 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\" (UniqueName: \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914211 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\" (UniqueName: \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914258 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914346 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914306 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914351 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:41.914649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914384 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\") pod \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\" (UID: \"2be5f650-8675-4ddf-9317-ec8a8cf1c780\") " Oct 30 20:31:41.914836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914765 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:41.914836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914791 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:41.915000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.914979 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:41.916739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.916715 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:41.917135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.917114 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:41.917252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:41.917183 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7" pod "2be5f650-8675-4ddf-9317-ec8a8cf1c780" (UID: "2be5f650-8675-4ddf-9317-ec8a8cf1c780"). InnerVolumeSpecName "kube-api-access-9pgz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.015823 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.015792 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.015994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.015847 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.015994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.015877 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.015994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.015926 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.015994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.015953 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.016192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016034 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.016192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016069 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.016192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016101 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.016192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016127 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.016192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016173 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\" (UniqueName: \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.016465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016209 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\") pod \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\" (UID: \"d9df96ea-61ae-44c6-8f37-e8fab797b53c\") " Oct 30 20:31:42.016465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016253 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\" (UniqueName: \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\") pod \"aaecb09e-0a11-4b21-b333-2f853774f3b5\" (UID: \"aaecb09e-0a11-4b21-b333-2f853774f3b5\") " Oct 30 20:31:42.016465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016249 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016751 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016784 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016802 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pgz7\" (UniqueName: \"kubernetes.io/projected/2be5f650-8675-4ddf-9317-ec8a8cf1c780-kube-api-access-9pgz7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016818 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2be5f650-8675-4ddf-9317-ec8a8cf1c780-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016833 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.016848 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2be5f650-8675-4ddf-9317-ec8a8cf1c780-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.017992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.017959 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.018119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.018041 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.018192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.018161 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.018657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.018436 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.018867 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.018804 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.018991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.018963 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "kube-api-access-n442h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.019411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.019388 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:42.019411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.019402 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm" pod "aaecb09e-0a11-4b21-b333-2f853774f3b5" (UID: "aaecb09e-0a11-4b21-b333-2f853774f3b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.020558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.020536 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "kube-api-access-nrbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:42.020983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.020958 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:42.021113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.021098 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm" pod "d9df96ea-61ae-44c6-8f37-e8fab797b53c" (UID: "d9df96ea-61ae-44c6-8f37-e8fab797b53c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117413 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117442 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tmp-dir\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117458 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbds\" (UniqueName: \"kubernetes.io/projected/d9df96ea-61ae-44c6-8f37-e8fab797b53c-kube-api-access-nrbds\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117470 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117483 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n442h\" (UniqueName: \"kubernetes.io/projected/aaecb09e-0a11-4b21-b333-2f853774f3b5-kube-api-access-n442h\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117512 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-model-cache\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117525 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaecb09e-0a11-4b21-b333-2f853774f3b5-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117536 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9df96ea-61ae-44c6-8f37-e8fab797b53c-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117546 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-home\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117558 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9df96ea-61ae-44c6-8f37-e8fab797b53c-tls-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.117846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.117569 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaecb09e-0a11-4b21-b333-2f853774f3b5-dshm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:42.672559 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.672522 4959 generic.go:356] "Generic (PLEG): container finished" podID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerID="5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c" exitCode=0 Oct 30 20:31:42.673015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.672589 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" Oct 30 20:31:42.673015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.672608 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" event={"ID":"d9df96ea-61ae-44c6-8f37-e8fab797b53c","Type":"ContainerDied","Data":"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c"} Oct 30 20:31:42.673015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.672651 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f" event={"ID":"d9df96ea-61ae-44c6-8f37-e8fab797b53c","Type":"ContainerDied","Data":"ee22f54c810011154a797ff88bf2c68eae50e3b1d56d6941236b470723ace608"} Oct 30 20:31:42.673015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.672674 4959 scope.go:122] "RemoveContainer" containerID="5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c" Oct 30 20:31:42.674169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.674144 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" Oct 30 20:31:42.674352 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.674302 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt" event={"ID":"aaecb09e-0a11-4b21-b333-2f853774f3b5","Type":"ContainerDied","Data":"6f7b309672a5c17c81532a4538d0ca52a818db39ca034b5da5f8a1fb781dccae"} Oct 30 20:31:42.675770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.675743 4959 generic.go:356] "Generic (PLEG): container finished" podID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerID="182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17" exitCode=0 Oct 30 20:31:42.675897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.675777 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" Oct 30 20:31:42.675897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.675780 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" event={"ID":"2be5f650-8675-4ddf-9317-ec8a8cf1c780","Type":"ContainerDied","Data":"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17"} Oct 30 20:31:42.676022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.675905 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw" event={"ID":"2be5f650-8675-4ddf-9317-ec8a8cf1c780","Type":"ContainerDied","Data":"ca9d5d1c49fe2bfd591dd048dbe6cdb0ba2b5430270cee11e884d0b05d6004a2"} Oct 30 20:31:42.683616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.683599 4959 scope.go:122] "RemoveContainer" containerID="5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c" Oct 30 20:31:42.683891 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:42.683872 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c\": container with ID starting with 5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c not found: ID does not exist" containerID="5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c" Oct 30 20:31:42.683966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.683898 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c"} err="failed to get container status \"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c\": rpc error: code = NotFound desc = could not find container \"5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c\": container with ID starting with 5f5e8cff7cd65acd6252f8ee77aee911d191d989c8c5ec58a8d1f84ce452375c not found: ID does not exist" Oct 30 20:31:42.683966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.683915 4959 scope.go:122] "RemoveContainer" containerID="ed86fb8ddecc6706f5ab99bebd27a2f344b95eb3935fcc156c13dee21b7fb59e" Oct 30 20:31:42.692551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.692530 4959 scope.go:122] "RemoveContainer" containerID="182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17" Oct 30 20:31:42.701128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.701101 4959 scope.go:122] "RemoveContainer" containerID="182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17" Oct 30 20:31:42.701423 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:42.701404 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17\": container with ID starting with 182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17 not found: ID does not exist" containerID="182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17" Oct 30 20:31:42.701486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.701431 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17"} err="failed to get container status \"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17\": rpc error: code = NotFound desc = could not find container \"182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17\": container with ID starting with 182dfc2fd62e14ede1bdd65c480e042e66b01ea3757714967fee38d83c617f17 not found: ID does not exist" Oct 30 20:31:42.735152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.735117 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 20:31:42.746660 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.746634 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-w6-f349a1-kserve-cc965856f-f6rsw"] Oct 30 20:31:42.796984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.796918 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 20:31:42.808319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.808230 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/premium-w6-f349a1-kserve-8585c86b7d-6572f"] Oct 30 20:31:42.844369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.844339 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 20:31:42.854873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:42.854848 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f349a1/distinct-2-w6-f349a1-kserve-6c765f6b66-czkgt"] Oct 30 20:31:43.213150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.213115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:43.416546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.415615 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:43.567567 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.551361 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 20:31:43.588857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.588804 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" containerID="cri-o://fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd" gracePeriod=30 Oct 30 20:31:43.613228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.613180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:43.994411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.994375 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" path="/var/lib/kubelet/pods/2be5f650-8675-4ddf-9317-ec8a8cf1c780/volumes" Oct 30 20:31:43.994851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.994788 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" path="/var/lib/kubelet/pods/aaecb09e-0a11-4b21-b333-2f853774f3b5/volumes" Oct 30 20:31:43.995137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:43.995123 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" path="/var/lib/kubelet/pods/d9df96ea-61ae-44c6-8f37-e8fab797b53c/volumes" Oct 30 20:31:44.082021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:44.081981 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:44.082021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:44.082007 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:45.304108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:45.304063 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:45.414824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:45.414765 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:45.673667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:45.673627 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 20:31:45.673937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:45.673905 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" containerID="cri-o://a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af" gracePeriod=30 Oct 30 20:31:45.707667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:45.707627 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:46.065272 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:46.065189 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:31:48.881803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.881746 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 20:31:48.987191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987158 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987206 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987233 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987259 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987330 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987367 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987402 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987434 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987455 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\") pod \"ed746e9e-a437-4203-98a7-15c919925e70\" (UID: \"ed746e9e-a437-4203-98a7-15c919925e70\") " Oct 30 20:31:48.987672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987485 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:48.987882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987741 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:48.987882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987760 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:48.987882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987772 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:48.987882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987749 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:48.988047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.987898 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:48.989894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.989869 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:48.990167 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.990137 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:48.990289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.990170 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:48.990289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:48.990202 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf" pod "ed746e9e-a437-4203-98a7-15c919925e70" (UID: "ed746e9e-a437-4203-98a7-15c919925e70"). InnerVolumeSpecName "kube-api-access-jpjtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:49.089113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089084 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089109 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjtf\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-kube-api-access-jpjtf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089119 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed746e9e-a437-4203-98a7-15c919925e70-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089127 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed746e9e-a437-4203-98a7-15c919925e70-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089136 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089143 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089151 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed746e9e-a437-4203-98a7-15c919925e70-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.089324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.089158 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed746e9e-a437-4203-98a7-15c919925e70-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:49.337913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.337876 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:49.716766 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.716727 4959 generic.go:356] "Generic (PLEG): container finished" podID="ed746e9e-a437-4203-98a7-15c919925e70" containerID="fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd" exitCode=0 Oct 30 20:31:49.716957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.716793 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" Oct 30 20:31:49.716957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.716829 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" event={"ID":"ed746e9e-a437-4203-98a7-15c919925e70","Type":"ContainerDied","Data":"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd"} Oct 30 20:31:49.716957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.716907 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj" event={"ID":"ed746e9e-a437-4203-98a7-15c919925e70","Type":"ContainerDied","Data":"4114e66d9dc5549b474ba09038476889bf3d53f916560f484f8ddf727d436355"} Oct 30 20:31:49.716957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.716950 4959 scope.go:122] "RemoveContainer" containerID="fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd" Oct 30 20:31:49.727234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.727217 4959 scope.go:122] "RemoveContainer" containerID="fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd" Oct 30 20:31:49.727572 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:49.727549 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd\": container with ID starting with fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd not found: ID does not exist" containerID="fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd" Oct 30 20:31:49.727687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.727580 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd"} err="failed to get container status \"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd\": rpc error: code = NotFound desc = could not find container \"fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd\": container with ID starting with fa05125167f0a671d608372bf3099351265af6c721d3534724de072378bdb9dd not found: ID does not exist" Oct 30 20:31:49.776367 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.776337 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 20:31:49.787542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.787520 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5db436-openshift-default-667b9cb645-bcgwj"] Oct 30 20:31:49.837924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.837897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:49.944787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.944756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:49.962864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.962821 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw"] Oct 30 20:31:49.964142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.964116 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f349a1-bxqqw"] Oct 30 20:31:49.996803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.996734 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33039e57-f4c9-4e41-ade3-7b7d44ec8583" path="/var/lib/kubelet/pods/33039e57-f4c9-4e41-ade3-7b7d44ec8583/volumes" Oct 30 20:31:49.997169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:49.997157 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed746e9e-a437-4203-98a7-15c919925e70" path="/var/lib/kubelet/pods/ed746e9e-a437-4203-98a7-15c919925e70/volumes" Oct 30 20:31:50.423990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:50.423936 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:50.535336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:50.535296 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:50.724947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:50.724914 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:50.923785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:50.923762 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 20:31:51.006263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006184 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006245 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006283 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006329 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006376 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006417 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006453 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006515 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006547 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\") pod \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\" (UID: \"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a\") " Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006561 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006576 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:51.006785 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006686 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:51.007522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006832 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:51.007522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006889 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.007522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006911 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.007522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006925 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.007522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.006980 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:51.009140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.009093 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "kube-api-access-9pgjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:51.009305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.009171 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:31:51.009378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.009331 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:31:51.009470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.009447 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token" pod "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" (UID: "d379af6f-5f5c-4a9f-abcb-e14e5fdb044a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:51.108419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108385 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.108419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108411 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.108419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108420 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.108419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108429 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.108725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108438 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.108725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.108446 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pgjm\" (UniqueName: \"kubernetes.io/projected/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a-kube-api-access-9pgjm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:51.210843 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.210809 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.316946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.316860 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.729904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.729869 4959 generic.go:356] "Generic (PLEG): container finished" podID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerID="a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af" exitCode=0 Oct 30 20:31:51.730092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.729939 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" Oct 30 20:31:51.730092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.729953 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerDied","Data":"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af"} Oct 30 20:31:51.730092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.730003 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g" event={"ID":"d379af6f-5f5c-4a9f-abcb-e14e5fdb044a","Type":"ContainerDied","Data":"26133b90ed2cebc7be791e9b7a1e81f00d0f19a11e007370129bcc8b3a5ece3d"} Oct 30 20:31:51.730092 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.730025 4959 scope.go:122] "RemoveContainer" containerID="a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af" Oct 30 20:31:51.740048 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.740032 4959 scope.go:122] "RemoveContainer" containerID="db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9" Oct 30 20:31:51.748773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.748755 4959 scope.go:122] "RemoveContainer" containerID="a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af" Oct 30 20:31:51.749032 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:51.749014 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af\": container with ID starting with a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af not found: ID does not exist" containerID="a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af" Oct 30 20:31:51.749088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.749042 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af"} err="failed to get container status \"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af\": rpc error: code = NotFound desc = could not find container \"a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af\": container with ID starting with a78e6d7c5fb39f49ae7964539701ac5f4dd140dd81c633b0f6a5161b603c03af not found: ID does not exist" Oct 30 20:31:51.749088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.749062 4959 scope.go:122] "RemoveContainer" containerID="db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9" Oct 30 20:31:51.749272 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:51.749257 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9\": container with ID starting with db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9 not found: ID does not exist" containerID="db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9" Oct 30 20:31:51.749313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.749275 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9"} err="failed to get container status \"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9\": rpc error: code = NotFound desc = could not find container \"db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9\": container with ID starting with db671a94d620f00551c97e03c9a7e7a74d63a3427dc9fa3fc59270ddc28c7bd9 not found: ID does not exist" Oct 30 20:31:51.789610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.789582 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 20:31:51.800653 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.800628 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-109cfe-openshift-default-5d45699b7f-64s4g"] Oct 30 20:31:51.995371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:51.995292 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" path="/var/lib/kubelet/pods/d379af6f-5f5c-4a9f-abcb-e14e5fdb044a/volumes" Oct 30 20:31:52.199109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:52.199070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:54.080137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:54.080106 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:54.080572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:54.080109 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:31:54.510948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:54.510914 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:54.618467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:54.618423 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:57.093337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.093309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:57.294287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.294265 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 20:31:57.306335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.306314 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 20:31:57.365063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365034 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\" (UniqueName: \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\") pod \"1c14d5fa-3809-4428-8be6-0a5501383465\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " Oct 30 20:31:57.365238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365074 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\") pod \"1c14d5fa-3809-4428-8be6-0a5501383465\" (UID: \"1c14d5fa-3809-4428-8be6-0a5501383465\") " Oct 30 20:31:57.365238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365103 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\" (UniqueName: \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\") pod \"151b789c-59c3-4f70-8b36-d7203dbed073\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " Oct 30 20:31:57.365238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365150 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\") pod \"151b789c-59c3-4f70-8b36-d7203dbed073\" (UID: \"151b789c-59c3-4f70-8b36-d7203dbed073\") " Oct 30 20:31:57.365623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365595 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume" pod "1c14d5fa-3809-4428-8be6-0a5501383465" (UID: "1c14d5fa-3809-4428-8be6-0a5501383465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:57.365711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.365640 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume" pod "151b789c-59c3-4f70-8b36-d7203dbed073" (UID: "151b789c-59c3-4f70-8b36-d7203dbed073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:57.368053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.368027 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px" pod "1c14d5fa-3809-4428-8be6-0a5501383465" (UID: "1c14d5fa-3809-4428-8be6-0a5501383465"). InnerVolumeSpecName "kube-api-access-hs5px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:57.368151 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.368057 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6" pod "151b789c-59c3-4f70-8b36-d7203dbed073" (UID: "151b789c-59c3-4f70-8b36-d7203dbed073"). InnerVolumeSpecName "kube-api-access-x8fq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:57.466783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.466747 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5px\" (UniqueName: \"kubernetes.io/projected/1c14d5fa-3809-4428-8be6-0a5501383465-kube-api-access-hs5px\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:57.466783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.466776 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c14d5fa-3809-4428-8be6-0a5501383465-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:57.466783 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.466786 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8fq6\" (UniqueName: \"kubernetes.io/projected/151b789c-59c3-4f70-8b36-d7203dbed073-kube-api-access-x8fq6\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:57.467006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.466796 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/151b789c-59c3-4f70-8b36-d7203dbed073-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:31:57.676403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.676358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:57.758314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.758280 4959 generic.go:356] "Generic (PLEG): container finished" podID="1c14d5fa-3809-4428-8be6-0a5501383465" containerID="c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad" exitCode=0 Oct 30 20:31:57.758565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.758331 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" Oct 30 20:31:57.758565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.758368 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" event={"ID":"1c14d5fa-3809-4428-8be6-0a5501383465","Type":"ContainerDied","Data":"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad"} Oct 30 20:31:57.758565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.758436 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw" event={"ID":"1c14d5fa-3809-4428-8be6-0a5501383465","Type":"ContainerDied","Data":"cea73a4249f04061b184890b7ca3949d79935aaee02e187bcf669e23e32bd007"} Oct 30 20:31:57.758565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.758470 4959 scope.go:122] "RemoveContainer" containerID="c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad" Oct 30 20:31:57.759769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.759749 4959 generic.go:356] "Generic (PLEG): container finished" podID="151b789c-59c3-4f70-8b36-d7203dbed073" containerID="496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690" exitCode=0 Oct 30 20:31:57.759869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.759797 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" Oct 30 20:31:57.759869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.759845 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" event={"ID":"151b789c-59c3-4f70-8b36-d7203dbed073","Type":"ContainerDied","Data":"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690"} Oct 30 20:31:57.759982 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.759886 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8" event={"ID":"151b789c-59c3-4f70-8b36-d7203dbed073","Type":"ContainerDied","Data":"7868caaa6b96234e92cf4a6b045faee334c397a73f78360fbb095d0d7a2a9944"} Oct 30 20:31:57.768341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.768321 4959 scope.go:122] "RemoveContainer" containerID="c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad" Oct 30 20:31:57.768651 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:57.768626 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad\": container with ID starting with c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad not found: ID does not exist" containerID="c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad" Oct 30 20:31:57.768769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.768655 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad"} err="failed to get container status \"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad\": rpc error: code = NotFound desc = could not find container \"c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad\": container with ID starting with c905ff38ec90f44d4b66e237917423f9a4643eed9d1a1258321d49608d9b7bad not found: ID does not exist" Oct 30 20:31:57.768769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.768672 4959 scope.go:122] "RemoveContainer" containerID="496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690" Oct 30 20:31:57.778182 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.778168 4959 scope.go:122] "RemoveContainer" containerID="496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690" Oct 30 20:31:57.778470 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:31:57.778453 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690\": container with ID starting with 496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690 not found: ID does not exist" containerID="496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690" Oct 30 20:31:57.778534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.778477 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690"} err="failed to get container status \"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690\": rpc error: code = NotFound desc = could not find container \"496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690\": container with ID starting with 496bc3eb8b2754c559e86163674eae76e69459605da16310bf24295502414690 not found: ID does not exist" Oct 30 20:31:57.780628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.780608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:57.804610 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.804589 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 20:31:57.808508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.808477 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f349a1-d8789d6cd-tbmxw"] Oct 30 20:31:57.840594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.840527 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 20:31:57.844342 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.844317 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f349a1-6c46dc8b59-5w4p8"] Oct 30 20:31:57.995601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.995515 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" path="/var/lib/kubelet/pods/151b789c-59c3-4f70-8b36-d7203dbed073/volumes" Oct 30 20:31:57.995897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:57.995882 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" path="/var/lib/kubelet/pods/1c14d5fa-3809-4428-8be6-0a5501383465/volumes" Oct 30 20:31:58.017203 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:58.017175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:59.038420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:31:59.038387 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:03.034828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:03.034791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:03.201998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:03.201959 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:03.307664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:03.307563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:04.889887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:04.889854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:10.759865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:10.759824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:14.928883 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:14.928842 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:16.065050 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:32:16.065016 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:32:24.738381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738341 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56"] Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738830 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738845 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738862 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738867 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738873 4959 cpu_manager.go:414] "containerMap: removing container" podUID="527f1372-f22d-450b-bca2-09d79e750070" containerName="maas-api" Oct 30 20:32:24.738878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738878 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527f1372-f22d-450b-bca2-09d79e750070" containerName="maas-api" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738884 4959 cpu_manager.go:414] "containerMap: removing container" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738889 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738899 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738903 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738913 4959 cpu_manager.go:414] "containerMap: removing container" podUID="39fe0673-9ebd-4de5-a67c-6828c712a279" containerName="revoke-keys" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738936 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fe0673-9ebd-4de5-a67c-6828c712a279" containerName="revoke-keys" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738986 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.738994 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739015 4959 cpu_manager.go:414] "containerMap: removing container" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739022 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739034 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739041 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739050 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" Oct 30 20:32:24.739091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739057 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739153 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c14d5fa-3809-4428-8be6-0a5501383465" containerName="payload-processing" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739166 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fe0673-9ebd-4de5-a67c-6828c712a279" containerName="revoke-keys" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739179 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739189 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="151b789c-59c3-4f70-8b36-d7203dbed073" containerName="payload-pre-processing" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739197 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9df96ea-61ae-44c6-8f37-e8fab797b53c" containerName="main" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739248 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed746e9e-a437-4203-98a7-15c919925e70" containerName="istio-proxy" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739262 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be5f650-8675-4ddf-9317-ec8a8cf1c780" containerName="main" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739276 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d379af6f-5f5c-4a9f-abcb-e14e5fdb044a" containerName="istio-proxy" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739286 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="527f1372-f22d-450b-bca2-09d79e750070" containerName="maas-api" Oct 30 20:32:24.739722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.739295 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaecb09e-0a11-4b21-b333-2f853774f3b5" containerName="main" Oct 30 20:32:24.742635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.742619 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.756581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.756563 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:32:24.767412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.767365 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:32:24.767575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.767529 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:32:24.767665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.767582 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:32:24.768921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.768896 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56"] Oct 30 20:32:24.811245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.811211 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfxn\" (UniqueName: \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.811423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.811265 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.811423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.811351 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.912120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.912082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfxn\" (UniqueName: \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.912347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.912140 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.912347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.912174 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.912901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.912876 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.914738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.914716 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:24.930915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:24.930890 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfxn\" (UniqueName: \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\") pod \"maas-api-revoke-sub-85237e45-bqh56\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:25.055114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:25.055033 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:25.351315 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:25.351201 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56"] Oct 30 20:32:25.356746 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:32:25.356717 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7e0280_0744_4d9e_944d_4e3d3c02bc71.slice/crio-3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e WatchSource:0}: Error finding container 3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e: Status 404 returned error can't find the container with id 3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e Oct 30 20:32:25.892179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:25.892140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" event={"ID":"9a7e0280-0744-4d9e-944d-4e3d3c02bc71","Type":"ContainerStarted","Data":"c679c41c1f82122f3a058da88a8242a7775338d4eaaf4b1214cbe5d0896a200d"} Oct 30 20:32:25.892618 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:25.892196 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" event={"ID":"9a7e0280-0744-4d9e-944d-4e3d3c02bc71","Type":"ContainerStarted","Data":"3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e"} Oct 30 20:32:26.898735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:26.898703 4959 generic.go:356] "Generic (PLEG): container finished" podID="9a7e0280-0744-4d9e-944d-4e3d3c02bc71" containerID="c679c41c1f82122f3a058da88a8242a7775338d4eaaf4b1214cbe5d0896a200d" exitCode=0 Oct 30 20:32:26.899137 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:26.898793 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" event={"ID":"9a7e0280-0744-4d9e-944d-4e3d3c02bc71","Type":"ContainerDied","Data":"c679c41c1f82122f3a058da88a8242a7775338d4eaaf4b1214cbe5d0896a200d"} Oct 30 20:32:28.037125 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.037099 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:28.140587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.140558 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\") pod \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " Oct 30 20:32:28.140784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.140614 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\" (UniqueName: \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\") pod \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " Oct 30 20:32:28.140784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.140650 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\") pod \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\" (UID: \"9a7e0280-0744-4d9e-944d-4e3d3c02bc71\") " Oct 30 20:32:28.141069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.141047 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca" pod "9a7e0280-0744-4d9e-944d-4e3d3c02bc71" (UID: "9a7e0280-0744-4d9e-944d-4e3d3c02bc71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:28.142696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.142678 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca" pod "9a7e0280-0744-4d9e-944d-4e3d3c02bc71" (UID: "9a7e0280-0744-4d9e-944d-4e3d3c02bc71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:28.142825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.142804 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn" pod "9a7e0280-0744-4d9e-944d-4e3d3c02bc71" (UID: "9a7e0280-0744-4d9e-944d-4e3d3c02bc71"). InnerVolumeSpecName "kube-api-access-jlfxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:28.242013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.241937 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:32:28.242013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.241961 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlfxn\" (UniqueName: \"kubernetes.io/projected/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-kube-api-access-jlfxn\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:32:28.242013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.241972 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7e0280-0744-4d9e-944d-4e3d3c02bc71-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:32:28.909796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.909770 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" Oct 30 20:32:28.910007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.909819 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56" event={"ID":"9a7e0280-0744-4d9e-944d-4e3d3c02bc71","Type":"ContainerDied","Data":"3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e"} Oct 30 20:32:28.910007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:28.909875 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4e69d9c5223d46cfb0f4c8732287c1acd465e28efaf9bd41b6339c1aa63a2e" Oct 30 20:32:30.451942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.451909 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt"] Oct 30 20:32:30.452456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.452426 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9a7e0280-0744-4d9e-944d-4e3d3c02bc71" containerName="revoke-keys" Oct 30 20:32:30.452575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.452460 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7e0280-0744-4d9e-944d-4e3d3c02bc71" containerName="revoke-keys" Oct 30 20:32:30.452693 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.452612 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7e0280-0744-4d9e-944d-4e3d3c02bc71" containerName="revoke-keys" Oct 30 20:32:30.455933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.455912 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.465553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.465532 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk"] Oct 30 20:32:30.469336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.469318 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.475369 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.475297 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-94wtv\"" Oct 30 20:32:30.483950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.483916 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt"] Oct 30 20:32:30.492763 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.492734 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk"] Oct 30 20:32:30.562101 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562071 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562109 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562131 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562261 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562296 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bd228-4267-40d4-af2e-198251489fce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562321 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562394 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562351 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562425 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bd228-4267-40d4-af2e-198251489fce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562468 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562486 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562526 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562573 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c950f1-eea3-4c70-9cda-3640eb311eee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562606 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562625 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whv8x\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-kube-api-access-whv8x\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562642 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562727 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.562872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562777 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hwm\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-kube-api-access-w5hwm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.562872 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.562796 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.663686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663654 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bd228-4267-40d4-af2e-198251489fce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.663686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663688 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.663959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663704 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.663959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663722 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bd228-4267-40d4-af2e-198251489fce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.663959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663741 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.663959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663763 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.663959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663925 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.663981 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c950f1-eea3-4c70-9cda-3640eb311eee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664024 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664052 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whv8x\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-kube-api-access-whv8x\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664081 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664131 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664140 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664161 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hwm\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-kube-api-access-w5hwm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664174 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664194 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664233 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664323 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664370 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664574 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bd228-4267-40d4-af2e-198251489fce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.664921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664771 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.665251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.664960 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.665251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.665016 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.665251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.665101 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c950f1-eea3-4c70-9cda-3640eb311eee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.665438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.665249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.665580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.665551 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.667049 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.667017 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.667245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.667228 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.667245 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.667238 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bd228-4267-40d4-af2e-198251489fce-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.667638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.667612 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bd228-4267-40d4-af2e-198251489fce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.682560 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.682540 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.682658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.682620 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whv8x\" (UniqueName: \"kubernetes.io/projected/98c950f1-eea3-4c70-9cda-3640eb311eee-kube-api-access-whv8x\") pod \"maas-default-gateway-openshift-default-75bf867b6c-ss8wt\" (UID: \"98c950f1-eea3-4c70-9cda-3640eb311eee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.682929 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.682907 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.683365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.683341 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hwm\" (UniqueName: \"kubernetes.io/projected/0f8bd228-4267-40d4-af2e-198251489fce-kube-api-access-w5hwm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h4twk\" (UID: \"0f8bd228-4267-40d4-af2e-198251489fce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:30.768930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.768840 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:30.788083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:30.788060 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:31.099169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.099087 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk"] Oct 30 20:32:31.101413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.101391 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt"] Oct 30 20:32:31.104055 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:32:31.104027 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8bd228_4267_40d4_af2e_198251489fce.slice/crio-994bb3c003c3253f59448f85ff5ff3f575bd121e574282828b90579c58547383 WatchSource:0}: Error finding container 994bb3c003c3253f59448f85ff5ff3f575bd121e574282828b90579c58547383: Status 404 returned error can't find the container with id 994bb3c003c3253f59448f85ff5ff3f575bd121e574282828b90579c58547383 Oct 30 20:32:31.106145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.106114 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.106214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.106201 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.106255 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.106233 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.117720 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:32:31.117692 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c950f1_eea3_4c70_9cda_3640eb311eee.slice/crio-89ef3f2a7a591944d9c822a60b780a5b196cacc9f8765edf9c7a34c0c109efa5 WatchSource:0}: Error finding container 89ef3f2a7a591944d9c822a60b780a5b196cacc9f8765edf9c7a34c0c109efa5: Status 404 returned error can't find the container with id 89ef3f2a7a591944d9c822a60b780a5b196cacc9f8765edf9c7a34c0c109efa5 Oct 30 20:32:31.119958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.119925 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.120047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.119990 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.120047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.120025 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:32:31.925906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.925870 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" event={"ID":"0f8bd228-4267-40d4-af2e-198251489fce","Type":"ContainerStarted","Data":"7d162d31f8644305cbb43821eb51b8714533141b437fa4d203f2d1b3564e929c"} Oct 30 20:32:31.926402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.925923 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" event={"ID":"0f8bd228-4267-40d4-af2e-198251489fce","Type":"ContainerStarted","Data":"994bb3c003c3253f59448f85ff5ff3f575bd121e574282828b90579c58547383"} Oct 30 20:32:31.927541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.927517 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" event={"ID":"98c950f1-eea3-4c70-9cda-3640eb311eee","Type":"ContainerStarted","Data":"1afb97ed3c716c367663c505e4efeabd31309843fa351323baa0618ff33c6b8b"} Oct 30 20:32:31.927661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:31.927559 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" event={"ID":"98c950f1-eea3-4c70-9cda-3640eb311eee","Type":"ContainerStarted","Data":"89ef3f2a7a591944d9c822a60b780a5b196cacc9f8765edf9c7a34c0c109efa5"} Oct 30 20:32:32.029891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.029844 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podStartSLOduration=2.029830597 podStartE2EDuration="2.029830597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:32.028998487 +0000 UTC m=+3476.158232408" watchObservedRunningTime="2026-10-30 20:32:32.029830597 +0000 UTC m=+3476.159064518" Oct 30 20:32:32.030519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.030479 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podStartSLOduration=2.030471893 podStartE2EDuration="2.030471893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:31.9897657 +0000 UTC m=+3476.118999623" watchObservedRunningTime="2026-10-30 20:32:32.030471893 +0000 UTC m=+3476.159705814" Oct 30 20:32:32.769908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.769871 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:32.774636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.774607 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:32.774861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.774836 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:32.796299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.796275 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:32.796625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.796606 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:32.796768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:32.796745 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:33.773956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:33.773916 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:33.774642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:33.774608 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:33.788897 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:33.788862 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:33.789126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:33.789100 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:34.121643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.121520 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:34.627184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.627148 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:34.773961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.773922 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:34.774643 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.774610 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:34.789923 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.789888 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:34.790186 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:34.790160 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:35.772986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:35.772948 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:35.773323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:35.773292 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:35.794414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:35.794380 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:35.795063 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:35.795033 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:36.498302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:36.498267 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:36.777577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:36.776619 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:36.777577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:36.776698 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:36.793520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:36.791917 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:36.793520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:36.791993 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:37.769910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:37.769873 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 30 20:32:37.770455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:37.769924 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 30 20:32:37.788364 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:37.788321 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 20:32:37.788565 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:37.788369 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 20:32:38.040839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:38.040753 4959 scope.go:122] "RemoveContainer" containerID="cd61ddedc7f8ca9000c06ce54d108d0ca38de5738f34d869a434294d0efd1fce" Oct 30 20:32:38.369345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:38.369230 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:39.446936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.446901 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:39.447488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.447463 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:39.448549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.448489 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" Oct 30 20:32:39.455834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.455813 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:39.456189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.456158 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:39.457007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:32:39.456970 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" Oct 30 20:32:46.064662 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:32:46.064624 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:33:15.464956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:15.464924 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:16.064303 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:33:16.064260 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:33:23.109217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.109180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:23.757282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.757241 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv"] Oct 30 20:33:23.761280 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.761260 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.786213 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.786183 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:33:23.786377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.786258 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:33:23.786377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.786343 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:33:23.786377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.786345 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:33:23.787848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.787828 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv"] Oct 30 20:33:23.879199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.879171 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.879403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.879233 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.879403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.879308 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cc7n\" (UniqueName: \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.980402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.980371 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.980615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.980422 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.980615 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.980441 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cc7n\" (UniqueName: \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.981180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.981145 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:23.982986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:23.982968 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:24.001917 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:24.001891 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cc7n\" (UniqueName: \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\") pod \"maas-api-revoke-sub-c1e271a9-vtfgv\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:24.074005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:24.073943 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:24.369709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:24.369438 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv"] Oct 30 20:33:24.375032 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:33:24.375002 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3735022_3dec_4620_951d_b22d8af01861.slice/crio-ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b WatchSource:0}: Error finding container ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b: Status 404 returned error can't find the container with id ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b Oct 30 20:33:25.181383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:25.181343 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" event={"ID":"a3735022-3dec-4620-951d-b22d8af01861","Type":"ContainerStarted","Data":"bfada8d33964198cbb0a1694115a94c948b8091bf19872331bff276dc1abf184"} Oct 30 20:33:25.181545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:25.181413 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" event={"ID":"a3735022-3dec-4620-951d-b22d8af01861","Type":"ContainerStarted","Data":"ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b"} Oct 30 20:33:25.218549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:25.218488 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" podStartSLOduration=2.218476186 podStartE2EDuration="2.218476186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:25.218295613 +0000 UTC m=+3529.347529532" watchObservedRunningTime="2026-10-30 20:33:25.218476186 +0000 UTC m=+3529.347710106" Oct 30 20:33:26.189889 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:26.189855 4959 generic.go:356] "Generic (PLEG): container finished" podID="a3735022-3dec-4620-951d-b22d8af01861" containerID="bfada8d33964198cbb0a1694115a94c948b8091bf19872331bff276dc1abf184" exitCode=0 Oct 30 20:33:26.190348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:26.189942 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" event={"ID":"a3735022-3dec-4620-951d-b22d8af01861","Type":"ContainerDied","Data":"bfada8d33964198cbb0a1694115a94c948b8091bf19872331bff276dc1abf184"} Oct 30 20:33:27.331215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.331192 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:27.410973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.410941 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\" (UniqueName: \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\") pod \"a3735022-3dec-4620-951d-b22d8af01861\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " Oct 30 20:33:27.411165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.410991 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\") pod \"a3735022-3dec-4620-951d-b22d8af01861\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " Oct 30 20:33:27.411165 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.411032 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\") pod \"a3735022-3dec-4620-951d-b22d8af01861\" (UID: \"a3735022-3dec-4620-951d-b22d8af01861\") " Oct 30 20:33:27.411481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.411459 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca" pod "a3735022-3dec-4620-951d-b22d8af01861" (UID: "a3735022-3dec-4620-951d-b22d8af01861"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:33:27.413259 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.413233 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca" pod "a3735022-3dec-4620-951d-b22d8af01861" (UID: "a3735022-3dec-4620-951d-b22d8af01861"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:33:27.413383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.413279 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n" pod "a3735022-3dec-4620-951d-b22d8af01861" (UID: "a3735022-3dec-4620-951d-b22d8af01861"). InnerVolumeSpecName "kube-api-access-6cc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:27.511809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.511723 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3735022-3dec-4620-951d-b22d8af01861-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:27.511809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.511753 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cc7n\" (UniqueName: \"kubernetes.io/projected/a3735022-3dec-4620-951d-b22d8af01861-kube-api-access-6cc7n\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:27.511809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:27.511764 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3735022-3dec-4620-951d-b22d8af01861-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:28.201147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:28.201110 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" event={"ID":"a3735022-3dec-4620-951d-b22d8af01861","Type":"ContainerDied","Data":"ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b"} Oct 30 20:33:28.201147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:28.201130 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv" Oct 30 20:33:28.201359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:28.201158 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5118b291e33c35d91f36531819393283e0d5923862e4c40f4204e3c2e74c0b" Oct 30 20:33:30.919641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:30.919608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:34.246561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:34.246453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:43.597124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.597080 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84"] Oct 30 20:33:43.597828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.597634 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a3735022-3dec-4620-951d-b22d8af01861" containerName="revoke-keys" Oct 30 20:33:43.597828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.597656 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3735022-3dec-4620-951d-b22d8af01861" containerName="revoke-keys" Oct 30 20:33:43.597828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.597779 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3735022-3dec-4620-951d-b22d8af01861" containerName="revoke-keys" Oct 30 20:33:43.601350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.601327 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.618835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.618804 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84"] Oct 30 20:33:43.628590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.628555 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:33:43.628787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.628579 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:33:43.628787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.628641 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:33:43.628997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.628642 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:33:43.761794 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.761762 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.761975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.761827 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2dq\" (UniqueName: \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.761975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.761911 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.862479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.862396 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2dq\" (UniqueName: \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.862479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.862444 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.862762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.862527 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.863292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.863262 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.866373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.866346 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.881075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.881044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2dq\" (UniqueName: \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\") pod \"maas-api-revoke-sub-2e1f5b5f-rzr84\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:43.914269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:43.914246 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:44.218408 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:44.218370 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84"] Oct 30 20:33:44.222894 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:33:44.222869 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5315a8_33e4_4f93_b1b7_2685fb17e284.slice/crio-2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b WatchSource:0}: Error finding container 2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b: Status 404 returned error can't find the container with id 2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b Oct 30 20:33:44.276440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:44.276410 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" event={"ID":"eb5315a8-33e4-4f93-b1b7-2685fb17e284","Type":"ContainerStarted","Data":"2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b"} Oct 30 20:33:45.282669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.282628 4959 generic.go:356] "Generic (PLEG): container finished" podID="eb5315a8-33e4-4f93-b1b7-2685fb17e284" containerID="df75513d31fef1bf17d5c0d4622acf8ece956f9ed82b7f4c8ece8e4eccd41a7d" exitCode=0 Oct 30 20:33:45.283082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.282719 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" event={"ID":"eb5315a8-33e4-4f93-b1b7-2685fb17e284","Type":"ContainerDied","Data":"df75513d31fef1bf17d5c0d4622acf8ece956f9ed82b7f4c8ece8e4eccd41a7d"} Oct 30 20:33:45.676194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.676161 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv"] Oct 30 20:33:45.679992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.679963 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.709635 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.709601 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh"] Oct 30 20:33:45.713852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.713829 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv"] Oct 30 20:33:45.713984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.713962 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.727911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.727883 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh"] Oct 30 20:33:45.781574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781535 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781586 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781684 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781757 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781747 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhz9\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-kube-api-access-wlhz9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781813 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781866 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.781937 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781913 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6676ac-0454-4b35-92b0-0969d06bb034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.782043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781937 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.782043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.781955 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883135 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883103 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883141 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883166 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883284 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhz9\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-kube-api-access-wlhz9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883344 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883378 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883402 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grb6\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-kube-api-access-6grb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883424 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883452 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883536 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883596 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883584 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6676ac-0454-4b35-92b0-0969d06bb034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883620 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883623 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883650 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883692 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883718 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883755 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883772 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883787 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.883966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.883874 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.884335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.884002 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.884335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.884275 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.884467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.884432 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c6676ac-0454-4b35-92b0-0969d06bb034-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.886392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.886361 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.886676 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.886657 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.902133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.902104 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.902411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.902394 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhz9\" (UniqueName: \"kubernetes.io/projected/0c6676ac-0454-4b35-92b0-0969d06bb034-kube-api-access-wlhz9\") pod \"maas-default-gateway-openshift-default-75bf867b6c-c69zv\" (UID: \"0c6676ac-0454-4b35-92b0-0969d06bb034\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:45.984947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.984868 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.984947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.984924 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985039 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grb6\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-kube-api-access-6grb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985111 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985185 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985142 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985282 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985317 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985322 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985437 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985479 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985586 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985825 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985761 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985835 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.985885 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.985858 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.987360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.987341 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.987702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.987684 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:45.994444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:45.994423 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:46.003477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.003454 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:46.003702 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.003677 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grb6\" (UniqueName: \"kubernetes.io/projected/5ba7dd9e-9da3-4885-b01a-4a942d81c097-kube-api-access-6grb6\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pq5nh\" (UID: \"5ba7dd9e-9da3-4885-b01a-4a942d81c097\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:46.032986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.032953 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:46.076369 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:33:46.076328 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:33:46.312416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.312379 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv"] Oct 30 20:33:46.316190 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:33:46.316164 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6676ac_0454_4b35_92b0_0969d06bb034.slice/crio-2c8dabfd13f3078226b5e51f765e47641f05012a03a7c03f89ac097e6fd9882b WatchSource:0}: Error finding container 2c8dabfd13f3078226b5e51f765e47641f05012a03a7c03f89ac097e6fd9882b: Status 404 returned error can't find the container with id 2c8dabfd13f3078226b5e51f765e47641f05012a03a7c03f89ac097e6fd9882b Oct 30 20:33:46.319737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.318634 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.319737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.318742 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.319737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.318785 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.357525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.354560 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh"] Oct 30 20:33:46.359144 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:33:46.359112 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba7dd9e_9da3_4885_b01a_4a942d81c097.slice/crio-5fec7b2107bd97953a0ed932e02c65f7ed0f7bf1ad947ea1c1c417eb444edd43 WatchSource:0}: Error finding container 5fec7b2107bd97953a0ed932e02c65f7ed0f7bf1ad947ea1c1c417eb444edd43: Status 404 returned error can't find the container with id 5fec7b2107bd97953a0ed932e02c65f7ed0f7bf1ad947ea1c1c417eb444edd43 Oct 30 20:33:46.361426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.361393 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.361511 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.361463 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.361562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.361518 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:33:46.424073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.424044 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:46.590166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.590048 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\" (UniqueName: \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\") pod \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " Oct 30 20:33:46.590166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.590098 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\") pod \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " Oct 30 20:33:46.590634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.590194 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\") pod \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\" (UID: \"eb5315a8-33e4-4f93-b1b7-2685fb17e284\") " Oct 30 20:33:46.590634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.590551 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca" pod "eb5315a8-33e4-4f93-b1b7-2685fb17e284" (UID: "eb5315a8-33e4-4f93-b1b7-2685fb17e284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:33:46.592528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.592464 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq" pod "eb5315a8-33e4-4f93-b1b7-2685fb17e284" (UID: "eb5315a8-33e4-4f93-b1b7-2685fb17e284"). InnerVolumeSpecName "kube-api-access-xk2dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:46.592899 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.592879 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca" pod "eb5315a8-33e4-4f93-b1b7-2685fb17e284" (UID: "eb5315a8-33e4-4f93-b1b7-2685fb17e284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:33:46.691358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.691320 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2dq\" (UniqueName: \"kubernetes.io/projected/eb5315a8-33e4-4f93-b1b7-2685fb17e284-kube-api-access-xk2dq\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:46.691358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.691349 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:46.691358 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:46.691358 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb5315a8-33e4-4f93-b1b7-2685fb17e284-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:47.304301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.304223 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" Oct 30 20:33:47.304508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.304254 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84" event={"ID":"eb5315a8-33e4-4f93-b1b7-2685fb17e284","Type":"ContainerDied","Data":"2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b"} Oct 30 20:33:47.304508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.304476 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2275e8f8d9a84f51f28268fbbc89c699de47967fa3858987eb8c2d32fa59a59b" Oct 30 20:33:47.305784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.305758 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" event={"ID":"0c6676ac-0454-4b35-92b0-0969d06bb034","Type":"ContainerStarted","Data":"6171f24866508605ea473e3377650153debbabbdca2c796ae9603a509c252856"} Oct 30 20:33:47.305933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.305801 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" event={"ID":"0c6676ac-0454-4b35-92b0-0969d06bb034","Type":"ContainerStarted","Data":"2c8dabfd13f3078226b5e51f765e47641f05012a03a7c03f89ac097e6fd9882b"} Oct 30 20:33:47.307776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.307699 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" event={"ID":"5ba7dd9e-9da3-4885-b01a-4a942d81c097","Type":"ContainerStarted","Data":"2ad9e5e04e22935a7bb439bd1fdb268fb9886129775feb4cc3fc4583fae215d2"} Oct 30 20:33:47.307776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.307746 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" event={"ID":"5ba7dd9e-9da3-4885-b01a-4a942d81c097","Type":"ContainerStarted","Data":"5fec7b2107bd97953a0ed932e02c65f7ed0f7bf1ad947ea1c1c417eb444edd43"} Oct 30 20:33:47.423430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.423376 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podStartSLOduration=2.423364245 podStartE2EDuration="2.423364245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:47.370754336 +0000 UTC m=+3551.499988258" watchObservedRunningTime="2026-10-30 20:33:47.423364245 +0000 UTC m=+3551.552598200" Oct 30 20:33:47.424237 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.423907 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podStartSLOduration=2.423898888 podStartE2EDuration="2.423898888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:47.421684628 +0000 UTC m=+3551.550918549" watchObservedRunningTime="2026-10-30 20:33:47.423898888 +0000 UTC m=+3551.553132802" Oct 30 20:33:47.998122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.998017 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:47.998122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:47.998078 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:48.002068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.002040 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:48.033931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.033905 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:48.034804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.034785 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:48.034908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.034833 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:48.997894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.997865 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:48.998359 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:48.998336 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:49.034120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:49.034072 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:49.034275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:49.034134 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:49.997895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:49.997858 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:49.998523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:49.998460 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:50.033673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:50.033639 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:50.033838 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:50.033700 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:50.998397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:50.997898 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:50.998397 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:50.997959 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:51.033713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:51.033681 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:51.033891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:51.033726 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:52.001521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.001149 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:52.001521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.001218 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:52.034076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.034037 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:52.034278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.034092 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:52.995389 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.995355 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 20:33:52.995575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:52.995417 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 20:33:53.033550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:53.033479 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 20:33:53.033940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:53.033579 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 20:33:54.721043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.721012 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:54.721989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.721646 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:54.722483 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.722463 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" Oct 30 20:33:54.726238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.726212 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:54.726611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.726592 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:54.727429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.727406 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" Oct 30 20:33:54.887229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.887195 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8"] Oct 30 20:33:54.888755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.888730 4959 cpu_manager.go:414] "containerMap: removing container" podUID="eb5315a8-33e4-4f93-b1b7-2685fb17e284" containerName="revoke-keys" Oct 30 20:33:54.888755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.888756 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb5315a8-33e4-4f93-b1b7-2685fb17e284" containerName="revoke-keys" Oct 30 20:33:54.888962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.888900 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb5315a8-33e4-4f93-b1b7-2685fb17e284" containerName="revoke-keys" Oct 30 20:33:54.892118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.892098 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:54.908405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.908378 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:33:54.908542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.908385 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:33:54.908718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.908702 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:33:54.908837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.908824 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:33:54.910526 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.910452 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8"] Oct 30 20:33:54.981630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.981551 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:54.981786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.981684 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klj8p\" (UniqueName: \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:54.981786 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:54.981721 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.082960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.082930 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.083124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.083007 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klj8p\" (UniqueName: \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.083124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.083028 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.083689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.083656 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.085376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.085354 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.100945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.100921 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klj8p\" (UniqueName: \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\") pod \"maas-api-revoke-sub-ec843018-jxsp8\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.209093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.209066 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:55.509292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:55.509200 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8"] Oct 30 20:33:55.514269 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:33:55.514243 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a16d503_2d9d_4314_acd3_ddcfddb7a837.slice/crio-6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3 WatchSource:0}: Error finding container 6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3: Status 404 returned error can't find the container with id 6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3 Oct 30 20:33:56.357955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:56.357863 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" event={"ID":"0a16d503-2d9d-4314-acd3-ddcfddb7a837","Type":"ContainerStarted","Data":"315804a1fcf03fa80a5257fed8b5cc47edbef09778b56cafc97b97a6974229ff"} Oct 30 20:33:56.357955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:56.357949 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" event={"ID":"0a16d503-2d9d-4314-acd3-ddcfddb7a837","Type":"ContainerStarted","Data":"6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3"} Oct 30 20:33:56.395029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:56.394595 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" podStartSLOduration=2.394578463 podStartE2EDuration="2.394578463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:56.393239561 +0000 UTC m=+3560.522473482" watchObservedRunningTime="2026-10-30 20:33:56.394578463 +0000 UTC m=+3560.523812385" Oct 30 20:33:57.383140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:57.383103 4959 generic.go:356] "Generic (PLEG): container finished" podID="0a16d503-2d9d-4314-acd3-ddcfddb7a837" containerID="315804a1fcf03fa80a5257fed8b5cc47edbef09778b56cafc97b97a6974229ff" exitCode=0 Oct 30 20:33:57.383782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:57.383755 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" event={"ID":"0a16d503-2d9d-4314-acd3-ddcfddb7a837","Type":"ContainerDied","Data":"315804a1fcf03fa80a5257fed8b5cc47edbef09778b56cafc97b97a6974229ff"} Oct 30 20:33:59.147034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.146745 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:59.217522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.216874 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\") pod \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " Oct 30 20:33:59.217522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.217052 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\" (UniqueName: \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\") pod \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " Oct 30 20:33:59.217522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.217116 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\") pod \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\" (UID: \"0a16d503-2d9d-4314-acd3-ddcfddb7a837\") " Oct 30 20:33:59.227536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.217889 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca" pod "0a16d503-2d9d-4314-acd3-ddcfddb7a837" (UID: "0a16d503-2d9d-4314-acd3-ddcfddb7a837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:33:59.230828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.230638 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p" pod "0a16d503-2d9d-4314-acd3-ddcfddb7a837" (UID: "0a16d503-2d9d-4314-acd3-ddcfddb7a837"). InnerVolumeSpecName "kube-api-access-klj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:59.239086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.239046 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca" pod "0a16d503-2d9d-4314-acd3-ddcfddb7a837" (UID: "0a16d503-2d9d-4314-acd3-ddcfddb7a837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:33:59.318767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.318674 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klj8p\" (UniqueName: \"kubernetes.io/projected/0a16d503-2d9d-4314-acd3-ddcfddb7a837-kube-api-access-klj8p\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:59.318767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.318710 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:59.318767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.318725 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a16d503-2d9d-4314-acd3-ddcfddb7a837-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:33:59.406741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.406714 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" Oct 30 20:33:59.406911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.406751 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8" event={"ID":"0a16d503-2d9d-4314-acd3-ddcfddb7a837","Type":"ContainerDied","Data":"6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3"} Oct 30 20:33:59.406911 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:33:59.406798 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fbe573b8ad3ac07d4da4e6203b524d521bce76ec21266d76e26fd7e85e76ad3" Oct 30 20:34:16.073633 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:34:16.073586 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:34:22.425507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.425467 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2"] Oct 30 20:34:22.426533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.426487 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0a16d503-2d9d-4314-acd3-ddcfddb7a837" containerName="revoke-keys" Oct 30 20:34:22.426533 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.426532 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a16d503-2d9d-4314-acd3-ddcfddb7a837" containerName="revoke-keys" Oct 30 20:34:22.427304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.426644 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a16d503-2d9d-4314-acd3-ddcfddb7a837" containerName="revoke-keys" Oct 30 20:34:22.430612 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.430595 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.455846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.455814 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:22.455846 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.455822 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:34:22.456056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.455825 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:34:22.456056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.455931 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:34:22.457705 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.457681 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2"] Oct 30 20:34:22.467030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.467003 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4phv\" (UniqueName: \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.467173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.467053 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.467173 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.467078 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.568343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.568309 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.568343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.568343 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.568587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.568427 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4phv\" (UniqueName: \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.569056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.569024 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.571035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.571010 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.586800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.586775 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4phv\" (UniqueName: \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\") pod \"maas-api-revoke-sub-046e1343-llsg2\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:22.745474 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:22.745406 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:23.041588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:23.041413 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2"] Oct 30 20:34:23.045722 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:34:23.045690 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e4f6f8_69a3_4167_855e_ecc41543847e.slice/crio-e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20 WatchSource:0}: Error finding container e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20: Status 404 returned error can't find the container with id e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20 Oct 30 20:34:23.584815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:23.584786 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" event={"ID":"e9e4f6f8-69a3-4167-855e-ecc41543847e","Type":"ContainerStarted","Data":"5a78e762d56d99ccbc0e15fb5a05ee54fa21a51017082ad814ab06eac58daa60"} Oct 30 20:34:23.585214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:23.584837 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" event={"ID":"e9e4f6f8-69a3-4167-855e-ecc41543847e","Type":"ContainerStarted","Data":"e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20"} Oct 30 20:34:23.623658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:23.623616 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" podStartSLOduration=1.6236046210000001 podStartE2EDuration="1.623604621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:23.622032905 +0000 UTC m=+3587.751266826" watchObservedRunningTime="2026-10-30 20:34:23.623604621 +0000 UTC m=+3587.752838539" Oct 30 20:34:24.590573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:24.590532 4959 generic.go:356] "Generic (PLEG): container finished" podID="e9e4f6f8-69a3-4167-855e-ecc41543847e" containerID="5a78e762d56d99ccbc0e15fb5a05ee54fa21a51017082ad814ab06eac58daa60" exitCode=0 Oct 30 20:34:24.590994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:24.590633 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" event={"ID":"e9e4f6f8-69a3-4167-855e-ecc41543847e","Type":"ContainerDied","Data":"5a78e762d56d99ccbc0e15fb5a05ee54fa21a51017082ad814ab06eac58daa60"} Oct 30 20:34:25.737401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.737376 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:25.795817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.795773 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\") pod \"e9e4f6f8-69a3-4167-855e-ecc41543847e\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " Oct 30 20:34:25.795987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.795909 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\" (UniqueName: \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\") pod \"e9e4f6f8-69a3-4167-855e-ecc41543847e\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " Oct 30 20:34:25.795987 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.795966 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\") pod \"e9e4f6f8-69a3-4167-855e-ecc41543847e\" (UID: \"e9e4f6f8-69a3-4167-855e-ecc41543847e\") " Oct 30 20:34:25.796141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.796114 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca" pod "e9e4f6f8-69a3-4167-855e-ecc41543847e" (UID: "e9e4f6f8-69a3-4167-855e-ecc41543847e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:25.796348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.796325 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:25.798247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.798220 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca" pod "e9e4f6f8-69a3-4167-855e-ecc41543847e" (UID: "e9e4f6f8-69a3-4167-855e-ecc41543847e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:25.798247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.798233 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv" pod "e9e4f6f8-69a3-4167-855e-ecc41543847e" (UID: "e9e4f6f8-69a3-4167-855e-ecc41543847e"). InnerVolumeSpecName "kube-api-access-k4phv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:25.897046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.897019 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4phv\" (UniqueName: \"kubernetes.io/projected/e9e4f6f8-69a3-4167-855e-ecc41543847e-kube-api-access-k4phv\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:25.897046 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:25.897045 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e4f6f8-69a3-4167-855e-ecc41543847e-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:26.600828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:26.600792 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" event={"ID":"e9e4f6f8-69a3-4167-855e-ecc41543847e","Type":"ContainerDied","Data":"e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20"} Oct 30 20:34:26.601010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:26.600850 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33807c1aff01c6d0aa46b0b4a1a0c8e43a92621ecf242cde81ab45db0720a20" Oct 30 20:34:26.601010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:26.600804 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2" Oct 30 20:34:32.785293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.785261 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6"] Oct 30 20:34:32.786336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.786310 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e9e4f6f8-69a3-4167-855e-ecc41543847e" containerName="revoke-keys" Oct 30 20:34:32.786336 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.786336 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e4f6f8-69a3-4167-855e-ecc41543847e" containerName="revoke-keys" Oct 30 20:34:32.786577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.786476 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e4f6f8-69a3-4167-855e-ecc41543847e" containerName="revoke-keys" Oct 30 20:34:32.789904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.789885 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.806864 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.806830 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:32.814966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.814943 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:34:32.815128 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.815074 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:34:32.815190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.815132 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:34:32.817134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.817107 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6"] Oct 30 20:34:32.857099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.857071 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.857287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.857105 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghtj\" (UniqueName: \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.857287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.857189 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.957731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.957702 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.957886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.957740 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghtj\" (UniqueName: \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.957886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.957788 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.958341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.958314 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.960138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.960118 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:32.976205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:32.976181 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghtj\" (UniqueName: \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\") pod \"maas-api-revoke-sub-d3fba056-tf2m6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:33.102761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:33.102701 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:33.403130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:33.403090 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6"] Oct 30 20:34:33.407968 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:34:33.407942 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6892e_2034_4a40_8037_bfe2ee16d1c6.slice/crio-98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637 WatchSource:0}: Error finding container 98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637: Status 404 returned error can't find the container with id 98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637 Oct 30 20:34:33.635599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:33.635568 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" event={"ID":"e0c6892e-2034-4a40-8037-bfe2ee16d1c6","Type":"ContainerStarted","Data":"98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637"} Oct 30 20:34:34.389127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:34.389095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:34.644700 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:34.644661 4959 generic.go:356] "Generic (PLEG): container finished" podID="e0c6892e-2034-4a40-8037-bfe2ee16d1c6" containerID="1c1371bd06242fab381e9ab70d5ed6b194b5889e144009cec6ade2a5de832de8" exitCode=0 Oct 30 20:34:34.644910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:34.644717 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" event={"ID":"e0c6892e-2034-4a40-8037-bfe2ee16d1c6","Type":"ContainerDied","Data":"1c1371bd06242fab381e9ab70d5ed6b194b5889e144009cec6ade2a5de832de8"} Oct 30 20:34:36.762016 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.761990 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:34:36.762888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.762861 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:34:36.768547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.768522 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:36.825415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.825368 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\") pod \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " Oct 30 20:34:36.825627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.825529 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\" (UniqueName: \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\") pod \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " Oct 30 20:34:36.825627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.825568 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\") pod \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\" (UID: \"e0c6892e-2034-4a40-8037-bfe2ee16d1c6\") " Oct 30 20:34:36.826202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.826174 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca" pod "e0c6892e-2034-4a40-8037-bfe2ee16d1c6" (UID: "e0c6892e-2034-4a40-8037-bfe2ee16d1c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:36.828750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.828721 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca" pod "e0c6892e-2034-4a40-8037-bfe2ee16d1c6" (UID: "e0c6892e-2034-4a40-8037-bfe2ee16d1c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:36.834948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.834919 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj" pod "e0c6892e-2034-4a40-8037-bfe2ee16d1c6" (UID: "e0c6892e-2034-4a40-8037-bfe2ee16d1c6"). InnerVolumeSpecName "kube-api-access-vghtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:36.926968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.926921 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghtj\" (UniqueName: \"kubernetes.io/projected/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-kube-api-access-vghtj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:36.926968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.926964 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:36.926968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:36.926975 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0c6892e-2034-4a40-8037-bfe2ee16d1c6-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:37.744111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:37.744085 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" Oct 30 20:34:37.744260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:37.744083 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6" event={"ID":"e0c6892e-2034-4a40-8037-bfe2ee16d1c6","Type":"ContainerDied","Data":"98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637"} Oct 30 20:34:37.744260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:37.744208 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f49c63f1f569050117248a2329f1f4aaf71f7a1843f511cb27e590fb770637" Oct 30 20:34:46.064778 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:34:46.064742 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:34:54.316468 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.316432 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l"] Oct 30 20:34:54.317171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.316892 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6892e-2034-4a40-8037-bfe2ee16d1c6" containerName="revoke-keys" Oct 30 20:34:54.317171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.316907 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6892e-2034-4a40-8037-bfe2ee16d1c6" containerName="revoke-keys" Oct 30 20:34:54.317171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.316992 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6892e-2034-4a40-8037-bfe2ee16d1c6" containerName="revoke-keys" Oct 30 20:34:54.319893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.319873 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.346674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.346642 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:34:54.346674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.346671 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:34:54.346873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.346648 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:34:54.346873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.346706 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:34:54.349145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.349119 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l"] Oct 30 20:34:54.432624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.432591 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5965h\" (UniqueName: \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.432826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.432693 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.432826 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.432808 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.533845 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.533809 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.534066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.533885 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5965h\" (UniqueName: \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.534066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.533916 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.534544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.534527 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.536382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.536361 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.553373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.553338 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5965h\" (UniqueName: \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\") pod \"maas-api-revoke-sub-ae594979-d594l\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.632850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.632826 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:54.930689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:54.930588 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l"] Oct 30 20:34:54.935420 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:34:54.935389 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc030a2_7c59_41f9_855e_93638b6d6e2f.slice/crio-60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7 WatchSource:0}: Error finding container 60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7: Status 404 returned error can't find the container with id 60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7 Oct 30 20:34:55.851809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:55.851772 4959 generic.go:356] "Generic (PLEG): container finished" podID="0cc030a2-7c59-41f9-855e-93638b6d6e2f" containerID="3712d8e66d3b0f8a7c223be5abc8d5da2989e32505e8aa604103a76a38eecad9" exitCode=0 Oct 30 20:34:55.852246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:55.851861 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" event={"ID":"0cc030a2-7c59-41f9-855e-93638b6d6e2f","Type":"ContainerDied","Data":"3712d8e66d3b0f8a7c223be5abc8d5da2989e32505e8aa604103a76a38eecad9"} Oct 30 20:34:55.852246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:55.851924 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" event={"ID":"0cc030a2-7c59-41f9-855e-93638b6d6e2f","Type":"ContainerStarted","Data":"60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7"} Oct 30 20:34:57.001029 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.001003 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:57.160109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.160085 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\") pod \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " Oct 30 20:34:57.160282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.160192 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\" (UniqueName: \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\") pod \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " Oct 30 20:34:57.160282 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.160225 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\") pod \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\" (UID: \"0cc030a2-7c59-41f9-855e-93638b6d6e2f\") " Oct 30 20:34:57.160663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.160640 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca" pod "0cc030a2-7c59-41f9-855e-93638b6d6e2f" (UID: "0cc030a2-7c59-41f9-855e-93638b6d6e2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:57.162319 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.162284 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca" pod "0cc030a2-7c59-41f9-855e-93638b6d6e2f" (UID: "0cc030a2-7c59-41f9-855e-93638b6d6e2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:57.162456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.162376 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h" pod "0cc030a2-7c59-41f9-855e-93638b6d6e2f" (UID: "0cc030a2-7c59-41f9-855e-93638b6d6e2f"). InnerVolumeSpecName "kube-api-access-5965h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:57.261252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.261225 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5965h\" (UniqueName: \"kubernetes.io/projected/0cc030a2-7c59-41f9-855e-93638b6d6e2f-kube-api-access-5965h\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:57.261252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.261247 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:57.261252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.261257 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc030a2-7c59-41f9-855e-93638b6d6e2f-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:34:57.863878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.863847 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" Oct 30 20:34:57.863878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.863869 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l" event={"ID":"0cc030a2-7c59-41f9-855e-93638b6d6e2f","Type":"ContainerDied","Data":"60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7"} Oct 30 20:34:57.864103 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:34:57.863924 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60c30c635f51887193954317c6b02708924ee48db913bfe86e873c689ad806d7" Oct 30 20:35:10.496746 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.496691 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h4twk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.497457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.497423 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h4twk" podUID="0f8bd228-4267-40d4-af2e-198251489fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:10.503752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.503689 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-ss8wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.504060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.504025 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-ss8wt" podUID="98c950f1-eea3-4c70-9cda-3640eb311eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:10.772579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.769682 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pq5nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.772579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.769754 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pq5nh" podUID="5ba7dd9e-9da3-4885-b01a-4a942d81c097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:10.784800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.773678 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-c69zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:35:10.784800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:10.773806 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-c69zv" podUID="0c6676ac-0454-4b35-92b0-0969d06bb034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:35:15.974335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.974300 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2"] Oct 30 20:35:15.974938 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.974918 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0cc030a2-7c59-41f9-855e-93638b6d6e2f" containerName="revoke-keys" Oct 30 20:35:15.975110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.974940 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc030a2-7c59-41f9-855e-93638b6d6e2f" containerName="revoke-keys" Oct 30 20:35:15.975110 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.975075 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc030a2-7c59-41f9-855e-93638b6d6e2f" containerName="revoke-keys" Oct 30 20:35:15.978426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.978405 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:15.992189 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:15.992161 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:35:16.003475 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.003448 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:35:16.003619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.003602 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:35:16.003741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.003715 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:35:16.005736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.005715 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2"] Oct 30 20:35:16.064839 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:35:16.064807 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:35:16.096422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.096395 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.096580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.096466 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.096580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.096567 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l859j\" (UniqueName: \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.197725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.197690 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.197898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.197768 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l859j\" (UniqueName: \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.197898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.197841 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.198666 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.198636 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.200129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.200111 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.216350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.216325 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l859j\" (UniqueName: \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\") pod \"maas-api-revoke-sub-a4aef7d7-z7qw2\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.291325 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.291242 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:16.586916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.586826 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2"] Oct 30 20:35:16.591949 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:35:16.591924 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e808dfe_e073_43c1_92f0_30885d1e7f0d.slice/crio-b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a WatchSource:0}: Error finding container b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a: Status 404 returned error can't find the container with id b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a Oct 30 20:35:16.971836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.971796 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" event={"ID":"1e808dfe-e073-43c1-92f0-30885d1e7f0d","Type":"ContainerStarted","Data":"40a9956930b9ab904a3d7929998ae83905076d84215987a14fb4bb548d531699"} Oct 30 20:35:16.971979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:16.971858 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" event={"ID":"1e808dfe-e073-43c1-92f0-30885d1e7f0d","Type":"ContainerStarted","Data":"b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a"} Oct 30 20:35:17.978038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:17.978005 4959 generic.go:356] "Generic (PLEG): container finished" podID="1e808dfe-e073-43c1-92f0-30885d1e7f0d" containerID="40a9956930b9ab904a3d7929998ae83905076d84215987a14fb4bb548d531699" exitCode=0 Oct 30 20:35:17.978459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:17.978090 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" event={"ID":"1e808dfe-e073-43c1-92f0-30885d1e7f0d","Type":"ContainerDied","Data":"40a9956930b9ab904a3d7929998ae83905076d84215987a14fb4bb548d531699"} Oct 30 20:35:19.132458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.132428 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:19.227829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.227805 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\") pod \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " Oct 30 20:35:19.228000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.227902 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\") pod \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " Oct 30 20:35:19.228000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.227947 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\" (UniqueName: \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\") pod \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\" (UID: \"1e808dfe-e073-43c1-92f0-30885d1e7f0d\") " Oct 30 20:35:19.228296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.228266 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca" pod "1e808dfe-e073-43c1-92f0-30885d1e7f0d" (UID: "1e808dfe-e073-43c1-92f0-30885d1e7f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:19.229943 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.229917 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca" pod "1e808dfe-e073-43c1-92f0-30885d1e7f0d" (UID: "1e808dfe-e073-43c1-92f0-30885d1e7f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:19.230100 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.230080 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j" pod "1e808dfe-e073-43c1-92f0-30885d1e7f0d" (UID: "1e808dfe-e073-43c1-92f0-30885d1e7f0d"). InnerVolumeSpecName "kube-api-access-l859j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:19.328570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.328543 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l859j\" (UniqueName: \"kubernetes.io/projected/1e808dfe-e073-43c1-92f0-30885d1e7f0d-kube-api-access-l859j\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:19.328570 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.328569 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:19.328772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.328578 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e808dfe-e073-43c1-92f0-30885d1e7f0d-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:19.989077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.989050 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" Oct 30 20:35:19.995280 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.995250 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2" event={"ID":"1e808dfe-e073-43c1-92f0-30885d1e7f0d","Type":"ContainerDied","Data":"b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a"} Oct 30 20:35:19.995430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:19.995292 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7729c6b992ea72d535841a0b55413fa8035813e54fba85ddd3a22741cb5709a" Oct 30 20:35:34.499257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:34.499225 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:37.083848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.083804 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn"] Oct 30 20:35:37.084732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.084707 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1e808dfe-e073-43c1-92f0-30885d1e7f0d" containerName="revoke-keys" Oct 30 20:35:37.084732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.084733 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e808dfe-e073-43c1-92f0-30885d1e7f0d" containerName="revoke-keys" Oct 30 20:35:37.084902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.084819 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e808dfe-e073-43c1-92f0-30885d1e7f0d" containerName="revoke-keys" Oct 30 20:35:37.088158 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.088139 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.104104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.104082 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:35:37.104222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.104101 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:35:37.104222 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.104128 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:35:37.115455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.115416 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:35:37.118622 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.118598 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn"] Oct 30 20:35:37.204928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.204899 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.205068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.204934 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.205068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.204956 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwk9p\" (UniqueName: \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.305857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.305831 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.305998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.305864 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.305998 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.305885 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwk9p\" (UniqueName: \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.306480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.306448 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.308365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.308346 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.325191 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.325159 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwk9p\" (UniqueName: \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\") pod \"maas-api-revoke-sub-79805ca9-dc2rn\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.400600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.400576 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:37.703403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:37.703320 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn"] Oct 30 20:35:37.707645 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:35:37.707622 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7832e75_938b_49e9_810e_17d8816e42dd.slice/crio-16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10 WatchSource:0}: Error finding container 16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10: Status 404 returned error can't find the container with id 16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10 Oct 30 20:35:38.088366 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:38.088331 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" event={"ID":"f7832e75-938b-49e9-810e-17d8816e42dd","Type":"ContainerStarted","Data":"6f2732686acf728eb1e113b8c01508509c87ae338d67fe54c676e56b9d326c7f"} Oct 30 20:35:38.088791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:38.088386 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" event={"ID":"f7832e75-938b-49e9-810e-17d8816e42dd","Type":"ContainerStarted","Data":"16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10"} Oct 30 20:35:38.127808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:38.127763 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" podStartSLOduration=1.127750431 podStartE2EDuration="1.127750431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:38.126353931 +0000 UTC m=+3662.255587853" watchObservedRunningTime="2026-10-30 20:35:38.127750431 +0000 UTC m=+3662.256984351" Oct 30 20:35:39.094209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:39.094176 4959 generic.go:356] "Generic (PLEG): container finished" podID="f7832e75-938b-49e9-810e-17d8816e42dd" containerID="6f2732686acf728eb1e113b8c01508509c87ae338d67fe54c676e56b9d326c7f" exitCode=0 Oct 30 20:35:39.094652 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:39.094274 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" event={"ID":"f7832e75-938b-49e9-810e-17d8816e42dd","Type":"ContainerDied","Data":"6f2732686acf728eb1e113b8c01508509c87ae338d67fe54c676e56b9d326c7f"} Oct 30 20:35:40.232099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.232070 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:40.331545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.331517 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\") pod \"f7832e75-938b-49e9-810e-17d8816e42dd\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " Oct 30 20:35:40.331695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.331580 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\" (UniqueName: \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\") pod \"f7832e75-938b-49e9-810e-17d8816e42dd\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " Oct 30 20:35:40.331695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.331618 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\") pod \"f7832e75-938b-49e9-810e-17d8816e42dd\" (UID: \"f7832e75-938b-49e9-810e-17d8816e42dd\") " Oct 30 20:35:40.332033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.332009 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca" pod "f7832e75-938b-49e9-810e-17d8816e42dd" (UID: "f7832e75-938b-49e9-810e-17d8816e42dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:35:40.333641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.333615 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca" pod "f7832e75-938b-49e9-810e-17d8816e42dd" (UID: "f7832e75-938b-49e9-810e-17d8816e42dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:35:40.333761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.333746 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p" pod "f7832e75-938b-49e9-810e-17d8816e42dd" (UID: "f7832e75-938b-49e9-810e-17d8816e42dd"). InnerVolumeSpecName "kube-api-access-xwk9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:35:40.432479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.432453 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:40.432479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.432476 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7832e75-938b-49e9-810e-17d8816e42dd-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:40.432654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:40.432485 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwk9p\" (UniqueName: \"kubernetes.io/projected/f7832e75-938b-49e9-810e-17d8816e42dd-kube-api-access-xwk9p\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:35:41.105563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:41.105536 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" Oct 30 20:35:41.105563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:41.105556 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn" event={"ID":"f7832e75-938b-49e9-810e-17d8816e42dd","Type":"ContainerDied","Data":"16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10"} Oct 30 20:35:41.105779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:41.105605 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16287e689ebec7d3910dc0d71b1b3d479cefd0f1915fc69b86e1dcb69bde5a10" Oct 30 20:35:46.065064 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:35:46.065024 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:35:57.108939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.108907 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq"] Oct 30 20:35:57.110111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.110081 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f7832e75-938b-49e9-810e-17d8816e42dd" containerName="revoke-keys" Oct 30 20:35:57.110111 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.110111 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7832e75-938b-49e9-810e-17d8816e42dd" containerName="revoke-keys" Oct 30 20:35:57.110275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.110244 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7832e75-938b-49e9-810e-17d8816e42dd" containerName="revoke-keys" Oct 30 20:35:57.116997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.116976 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.131528 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.131481 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq"] Oct 30 20:35:57.140461 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.140437 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:35:57.140598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.140545 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:35:57.141810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.141788 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:35:57.141810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.141796 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:35:57.215749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.215720 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.215934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.215756 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.215934 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.215827 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvlhr\" (UniqueName: \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.317235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.317200 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.317235 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.317241 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.317430 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.317273 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvlhr\" (UniqueName: \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.317973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.317953 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.319703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.319686 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.335650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.335621 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvlhr\" (UniqueName: \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\") pod \"maas-api-revoke-sub-0b7e8edb-9kxrq\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.429878 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.429853 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:35:57.727971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:57.727878 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq"] Oct 30 20:35:57.732630 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:35:57.732602 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c8da5a_9c30_4fb7_9e7e_94c3cc3cc5da.slice/crio-9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15 WatchSource:0}: Error finding container 9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15: Status 404 returned error can't find the container with id 9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15 Oct 30 20:35:58.226663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:58.226624 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" event={"ID":"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da","Type":"ContainerStarted","Data":"fc5986af3ca8d92d098b5616edefbc39a6a53207d9a42d49b56e2b065bfd2bc1"} Oct 30 20:35:58.227038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:58.226689 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" event={"ID":"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da","Type":"ContainerStarted","Data":"9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15"} Oct 30 20:35:58.264822 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:58.264780 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" podStartSLOduration=1.264769396 podStartE2EDuration="1.264769396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:58.263535392 +0000 UTC m=+3682.392769309" watchObservedRunningTime="2026-10-30 20:35:58.264769396 +0000 UTC m=+3682.394003316" Oct 30 20:35:59.232220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:59.232190 4959 generic.go:356] "Generic (PLEG): container finished" podID="56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" containerID="fc5986af3ca8d92d098b5616edefbc39a6a53207d9a42d49b56e2b065bfd2bc1" exitCode=0 Oct 30 20:35:59.232639 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:35:59.232280 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" event={"ID":"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da","Type":"ContainerDied","Data":"fc5986af3ca8d92d098b5616edefbc39a6a53207d9a42d49b56e2b065bfd2bc1"} Oct 30 20:36:00.382295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.382269 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:36:00.548465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.548372 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\") pod \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " Oct 30 20:36:00.548680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.548543 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\") pod \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " Oct 30 20:36:00.548680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.548572 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\" (UniqueName: \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\") pod \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\" (UID: \"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da\") " Oct 30 20:36:00.548815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.548771 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca" pod "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" (UID: "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:00.550870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.550842 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr" pod "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" (UID: "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da"). InnerVolumeSpecName "kube-api-access-gvlhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:00.550975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.550902 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca" pod "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" (UID: "56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:00.649904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.649869 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:00.649904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.649902 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvlhr\" (UniqueName: \"kubernetes.io/projected/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-kube-api-access-gvlhr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:00.650118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:00.649916 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:01.242948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:01.242916 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" Oct 30 20:36:01.243129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:01.242953 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq" event={"ID":"56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da","Type":"ContainerDied","Data":"9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15"} Oct 30 20:36:01.243129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:01.243000 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eac6c5cf777d39c47aaecc41d37d00560a5ea9e7c62c5f1bd0af2e15369ad15" Oct 30 20:36:07.864740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.864698 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54"] Oct 30 20:36:07.865196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.865174 4959 cpu_manager.go:414] "containerMap: removing container" podUID="56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" containerName="revoke-keys" Oct 30 20:36:07.865196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.865189 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" containerName="revoke-keys" Oct 30 20:36:07.865285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.865275 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" containerName="revoke-keys" Oct 30 20:36:07.868617 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.868592 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:07.885266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.885237 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:07.887555 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.887518 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54"] Oct 30 20:36:07.896752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.896720 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:36:07.896752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.896729 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:36:07.896752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:07.896752 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:36:08.026387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.026338 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bl7b\" (UniqueName: \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.026387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.026396 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.026672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.026461 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.127400 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.127311 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bl7b\" (UniqueName: \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.127400 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.127358 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.127400 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.127395 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.128184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.128160 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.129944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.129925 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.146261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.146227 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bl7b\" (UniqueName: \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\") pod \"maas-api-revoke-sub-3d4156f1-rdk54\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.182096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.182060 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:08.482787 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:08.482750 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54"] Oct 30 20:36:08.487308 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:36:08.487279 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62cfff3_dc78_47c3_b8d9_23fd119558db.slice/crio-94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834 WatchSource:0}: Error finding container 94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834: Status 404 returned error can't find the container with id 94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834 Oct 30 20:36:09.285065 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:09.285032 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" event={"ID":"d62cfff3-dc78-47c3-b8d9-23fd119558db","Type":"ContainerStarted","Data":"89c91b320e0489fc80d2633aa531821090a08f9b918c36c227a9839c4edc1e70"} Oct 30 20:36:09.285414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:09.285095 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" event={"ID":"d62cfff3-dc78-47c3-b8d9-23fd119558db","Type":"ContainerStarted","Data":"94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834"} Oct 30 20:36:10.291678 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.291642 4959 generic.go:356] "Generic (PLEG): container finished" podID="d62cfff3-dc78-47c3-b8d9-23fd119558db" containerID="89c91b320e0489fc80d2633aa531821090a08f9b918c36c227a9839c4edc1e70" exitCode=0 Oct 30 20:36:10.292132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.291686 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" event={"ID":"d62cfff3-dc78-47c3-b8d9-23fd119558db","Type":"ContainerDied","Data":"89c91b320e0489fc80d2633aa531821090a08f9b918c36c227a9839c4edc1e70"} Oct 30 20:36:10.435781 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.435753 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:10.550429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.550333 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\") pod \"d62cfff3-dc78-47c3-b8d9-23fd119558db\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " Oct 30 20:36:10.550429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.550405 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\" (UniqueName: \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\") pod \"d62cfff3-dc78-47c3-b8d9-23fd119558db\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " Oct 30 20:36:10.550717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.550509 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\") pod \"d62cfff3-dc78-47c3-b8d9-23fd119558db\" (UID: \"d62cfff3-dc78-47c3-b8d9-23fd119558db\") " Oct 30 20:36:10.550837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.550810 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca" pod "d62cfff3-dc78-47c3-b8d9-23fd119558db" (UID: "d62cfff3-dc78-47c3-b8d9-23fd119558db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:10.552799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.552773 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b" pod "d62cfff3-dc78-47c3-b8d9-23fd119558db" (UID: "d62cfff3-dc78-47c3-b8d9-23fd119558db"). InnerVolumeSpecName "kube-api-access-8bl7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:10.552799 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.552794 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca" pod "d62cfff3-dc78-47c3-b8d9-23fd119558db" (UID: "d62cfff3-dc78-47c3-b8d9-23fd119558db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:10.652229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.652192 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:10.652229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.652224 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bl7b\" (UniqueName: \"kubernetes.io/projected/d62cfff3-dc78-47c3-b8d9-23fd119558db-kube-api-access-8bl7b\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:10.652229 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:10.652234 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d62cfff3-dc78-47c3-b8d9-23fd119558db-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:11.298719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:11.298689 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" Oct 30 20:36:11.298719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:11.298700 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54" event={"ID":"d62cfff3-dc78-47c3-b8d9-23fd119558db","Type":"ContainerDied","Data":"94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834"} Oct 30 20:36:11.299164 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:11.298748 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e4791505995fe919213abaf3df19ab40db3e50fb673a71f677ea5c46f62834" Oct 30 20:36:16.065241 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:36:16.065199 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:36:29.207037 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.206994 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54"] Oct 30 20:36:29.217327 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.217301 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rdk54"] Oct 30 20:36:29.250927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.250896 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6"] Oct 30 20:36:29.251531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.251486 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d62cfff3-dc78-47c3-b8d9-23fd119558db" containerName="revoke-keys" Oct 30 20:36:29.251531 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.251530 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62cfff3-dc78-47c3-b8d9-23fd119558db" containerName="revoke-keys" Oct 30 20:36:29.251773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.251632 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62cfff3-dc78-47c3-b8d9-23fd119558db" containerName="revoke-keys" Oct 30 20:36:29.254761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.254740 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.271335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.271313 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:36:29.271551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.271525 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:36:29.271686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.271619 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:36:29.271773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.271722 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:36:29.274215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.273969 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6"] Oct 30 20:36:29.339140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.339112 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.339365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.339156 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.339365 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.339309 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9tk\" (UniqueName: \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.440673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.440637 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.440891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.440690 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.440891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.440745 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9tk\" (UniqueName: \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.441424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.441403 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.443317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.443298 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.459403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.459331 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9tk\" (UniqueName: \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\") pod \"maas-api-revoke-sub-3d4156f1-jvwh6\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.568318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.568290 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:29.873413 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.873340 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6"] Oct 30 20:36:29.877483 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:36:29.877454 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21581fc_5382_4489_9774_82b9a9d4f330.slice/crio-a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058 WatchSource:0}: Error finding container a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058: Status 404 returned error can't find the container with id a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058 Oct 30 20:36:29.995081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:29.995036 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62cfff3-dc78-47c3-b8d9-23fd119558db" path="/var/lib/kubelet/pods/d62cfff3-dc78-47c3-b8d9-23fd119558db/volumes" Oct 30 20:36:30.416234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:30.416140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" event={"ID":"c21581fc-5382-4489-9774-82b9a9d4f330","Type":"ContainerStarted","Data":"67fc4908ec1ef231300b7139c4dc4b275f589adbf539daa9a3f2de5b4f6b8e16"} Oct 30 20:36:30.416234 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:30.416200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" event={"ID":"c21581fc-5382-4489-9774-82b9a9d4f330","Type":"ContainerStarted","Data":"a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058"} Oct 30 20:36:30.453974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:30.453924 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" podStartSLOduration=1.453911999 podStartE2EDuration="1.453911999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:30.453699741 +0000 UTC m=+3714.582933662" watchObservedRunningTime="2026-10-30 20:36:30.453911999 +0000 UTC m=+3714.583145920" Oct 30 20:36:31.422748 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:31.422711 4959 generic.go:356] "Generic (PLEG): container finished" podID="c21581fc-5382-4489-9774-82b9a9d4f330" containerID="67fc4908ec1ef231300b7139c4dc4b275f589adbf539daa9a3f2de5b4f6b8e16" exitCode=0 Oct 30 20:36:31.423169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:31.422799 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" event={"ID":"c21581fc-5382-4489-9774-82b9a9d4f330","Type":"ContainerDied","Data":"67fc4908ec1ef231300b7139c4dc4b275f589adbf539daa9a3f2de5b4f6b8e16"} Oct 30 20:36:32.570884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.570859 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:32.674585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.674555 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\") pod \"c21581fc-5382-4489-9774-82b9a9d4f330\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " Oct 30 20:36:32.674796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.674610 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\" (UniqueName: \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\") pod \"c21581fc-5382-4489-9774-82b9a9d4f330\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " Oct 30 20:36:32.674796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.674720 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\") pod \"c21581fc-5382-4489-9774-82b9a9d4f330\" (UID: \"c21581fc-5382-4489-9774-82b9a9d4f330\") " Oct 30 20:36:32.675004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.674981 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca" pod "c21581fc-5382-4489-9774-82b9a9d4f330" (UID: "c21581fc-5382-4489-9774-82b9a9d4f330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:32.676866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.676846 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca" pod "c21581fc-5382-4489-9774-82b9a9d4f330" (UID: "c21581fc-5382-4489-9774-82b9a9d4f330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:36:32.677001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.676986 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk" pod "c21581fc-5382-4489-9774-82b9a9d4f330" (UID: "c21581fc-5382-4489-9774-82b9a9d4f330"). InnerVolumeSpecName "kube-api-access-df9tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:32.775625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.775589 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:32.775625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.775620 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9tk\" (UniqueName: \"kubernetes.io/projected/c21581fc-5382-4489-9774-82b9a9d4f330-kube-api-access-df9tk\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:32.775625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:32.775630 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21581fc-5382-4489-9774-82b9a9d4f330-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:36:33.434598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:33.434564 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" Oct 30 20:36:33.434791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:33.434569 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6" event={"ID":"c21581fc-5382-4489-9774-82b9a9d4f330","Type":"ContainerDied","Data":"a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058"} Oct 30 20:36:33.434791 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:33.434696 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2aa5708e3924caee89b96a30e11d1feb2dfa127921899b68bf288790960e058" Oct 30 20:36:34.594720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:34.594685 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:35.233739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:35.233704 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:39.032082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:39.032047 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99"] Oct 30 20:36:39.042903 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:39.042871 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff98e46-68r99"] Oct 30 20:36:39.995435 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:36:39.995399 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fe0673-9ebd-4de5-a67c-6828c712a279" path="/var/lib/kubelet/pods/39fe0673-9ebd-4de5-a67c-6828c712a279/volumes" Oct 30 20:36:46.064761 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:36:46.064728 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:37:02.961925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.961885 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s"] Oct 30 20:37:02.962960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.962531 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c21581fc-5382-4489-9774-82b9a9d4f330" containerName="revoke-keys" Oct 30 20:37:02.962960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.962553 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21581fc-5382-4489-9774-82b9a9d4f330" containerName="revoke-keys" Oct 30 20:37:02.962960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.962692 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21581fc-5382-4489-9774-82b9a9d4f330" containerName="revoke-keys" Oct 30 20:37:02.966268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.966247 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:02.982175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.982155 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:37:02.993216 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.993181 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:37:02.995391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:02.995361 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s"] Oct 30 20:37:03.029731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.029699 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:37:03.029890 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.029738 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:37:03.077668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.077634 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.077816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.077679 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.077816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.077782 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwqt\" (UniqueName: \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.178946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.178909 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.179132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.178960 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.179132 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.178992 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwqt\" (UniqueName: \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.179804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.179731 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.181593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.181565 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.197879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.197858 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwqt\" (UniqueName: \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\") pod \"maas-api-revoke-sub-e2fc9c5b-fzk8s\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.279147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.279066 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:03.578764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.578662 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s"] Oct 30 20:37:03.583464 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:37:03.583435 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d6ee3a_f460_4ceb_a3e8_a15584b9bf1b.slice/crio-bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac WatchSource:0}: Error finding container bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac: Status 404 returned error can't find the container with id bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac Oct 30 20:37:03.624962 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:03.624927 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" event={"ID":"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b","Type":"ContainerStarted","Data":"bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac"} Oct 30 20:37:04.461866 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:37:04.461839 4959 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d6ee3a_f460_4ceb_a3e8_a15584b9bf1b.slice/crio-conmon-0d7dd70e854bb5ace0f5cdd5f8ec43f232d4c19e96df9b8689ad7cd37e7e9e7e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:37:04.631362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:04.631276 4959 generic.go:356] "Generic (PLEG): container finished" podID="b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" containerID="0d7dd70e854bb5ace0f5cdd5f8ec43f232d4c19e96df9b8689ad7cd37e7e9e7e" exitCode=0 Oct 30 20:37:04.631546 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:04.631367 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" event={"ID":"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b","Type":"ContainerDied","Data":"0d7dd70e854bb5ace0f5cdd5f8ec43f232d4c19e96df9b8689ad7cd37e7e9e7e"} Oct 30 20:37:05.772628 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.772605 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:05.905431 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.905359 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\" (UniqueName: \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\") pod \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " Oct 30 20:37:05.905627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.905456 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\") pod \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " Oct 30 20:37:05.905627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.905483 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\") pod \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\" (UID: \"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b\") " Oct 30 20:37:05.905857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.905829 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca" pod "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" (UID: "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:05.907706 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.907679 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca" pod "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" (UID: "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:05.907824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:05.907779 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt" pod "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" (UID: "b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b"). InnerVolumeSpecName "kube-api-access-jpwqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:06.006314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.006295 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:06.006412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.006315 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:06.006412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.006326 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwqt\" (UniqueName: \"kubernetes.io/projected/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b-kube-api-access-jpwqt\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:06.643355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.643325 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" Oct 30 20:37:06.643574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.643372 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s" event={"ID":"b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b","Type":"ContainerDied","Data":"bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac"} Oct 30 20:37:06.643574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:06.643423 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf2713de54d96ac612952378cbf9ccf85ee46317687d6855b0f13a28c371faac" Oct 30 20:37:13.305384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.305346 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj"] Oct 30 20:37:13.306171 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.306145 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" containerName="revoke-keys" Oct 30 20:37:13.306356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.306333 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" containerName="revoke-keys" Oct 30 20:37:13.306562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.306546 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" containerName="revoke-keys" Oct 30 20:37:13.309754 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.309733 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.324104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.324071 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:37:13.324256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.324106 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:37:13.335384 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.335353 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:37:13.335593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.335416 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:37:13.337767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.337737 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj"] Oct 30 20:37:13.472715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.472675 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.472902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.472718 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.472902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.472834 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzz5n\" (UniqueName: \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.573458 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.573379 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzz5n\" (UniqueName: \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.573642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.573468 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.573642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.573488 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.574075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.574058 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.575956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.575936 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.592264 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.592240 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzz5n\" (UniqueName: \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\") pod \"maas-api-revoke-sub-5a29efb3-qk4lj\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.623067 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.623044 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:13.920892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:13.920854 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj"] Oct 30 20:37:13.925720 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:37:13.925690 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d36a77_e403_4214_8e87_877e94cf4c79.slice/crio-3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854 WatchSource:0}: Error finding container 3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854: Status 404 returned error can't find the container with id 3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854 Oct 30 20:37:14.687600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:14.687541 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" event={"ID":"44d36a77-e403-4214-8e87-877e94cf4c79","Type":"ContainerStarted","Data":"99fe084c2476bbdd4b2c3ac8eb17edf06c58b0a1ca33e9865fab942237cde292"} Oct 30 20:37:14.688106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:14.687639 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" event={"ID":"44d36a77-e403-4214-8e87-877e94cf4c79","Type":"ContainerStarted","Data":"3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854"} Oct 30 20:37:14.737776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:14.737701 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" podStartSLOduration=1.7376815890000001 podStartE2EDuration="1.737681589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:14.734741038 +0000 UTC m=+3758.863974961" watchObservedRunningTime="2026-10-30 20:37:14.737681589 +0000 UTC m=+3758.866915511" Oct 30 20:37:15.707406 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:15.707359 4959 generic.go:356] "Generic (PLEG): container finished" podID="44d36a77-e403-4214-8e87-877e94cf4c79" containerID="99fe084c2476bbdd4b2c3ac8eb17edf06c58b0a1ca33e9865fab942237cde292" exitCode=0 Oct 30 20:37:15.712290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:15.712256 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" event={"ID":"44d36a77-e403-4214-8e87-877e94cf4c79","Type":"ContainerDied","Data":"99fe084c2476bbdd4b2c3ac8eb17edf06c58b0a1ca33e9865fab942237cde292"} Oct 30 20:37:16.297852 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:37:16.297809 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:37:17.443698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.443671 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:17.515884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.515840 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\") pod \"44d36a77-e403-4214-8e87-877e94cf4c79\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " Oct 30 20:37:17.516590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.516565 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\" (UniqueName: \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\") pod \"44d36a77-e403-4214-8e87-877e94cf4c79\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " Oct 30 20:37:17.516820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.516802 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\") pod \"44d36a77-e403-4214-8e87-877e94cf4c79\" (UID: \"44d36a77-e403-4214-8e87-877e94cf4c79\") " Oct 30 20:37:17.517566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.517544 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca" pod "44d36a77-e403-4214-8e87-877e94cf4c79" (UID: "44d36a77-e403-4214-8e87-877e94cf4c79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:17.531642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.531617 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca" pod "44d36a77-e403-4214-8e87-877e94cf4c79" (UID: "44d36a77-e403-4214-8e87-877e94cf4c79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:17.531808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.531791 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n" pod "44d36a77-e403-4214-8e87-877e94cf4c79" (UID: "44d36a77-e403-4214-8e87-877e94cf4c79"). InnerVolumeSpecName "kube-api-access-wzz5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:17.618562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.618521 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:17.618562 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.618556 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzz5n\" (UniqueName: \"kubernetes.io/projected/44d36a77-e403-4214-8e87-877e94cf4c79-kube-api-access-wzz5n\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:17.618761 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.618572 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d36a77-e403-4214-8e87-877e94cf4c79-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:17.730551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.730463 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" Oct 30 20:37:17.730551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.730464 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj" event={"ID":"44d36a77-e403-4214-8e87-877e94cf4c79","Type":"ContainerDied","Data":"3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854"} Oct 30 20:37:17.730749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:17.730605 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c4ff3635c2e37a55312f14e3c001cbf09064ec04f74235df08b5b5147525854" Oct 30 20:37:23.979479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:23.979441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:24.086464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:24.086427 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:28.027969 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:28.027935 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56"] Oct 30 20:37:28.039117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:28.039089 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bqh56"] Oct 30 20:37:29.995467 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:29.995432 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7e0280-0744-4d9e-944d-4e3d3c02bc71" path="/var/lib/kubelet/pods/9a7e0280-0744-4d9e-944d-4e3d3c02bc71/volumes" Oct 30 20:37:32.476704 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.476667 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6"] Oct 30 20:37:32.487654 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.487624 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvwh6"] Oct 30 20:37:32.523720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.523683 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4"] Oct 30 20:37:32.524363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.524340 4959 cpu_manager.go:414] "containerMap: removing container" podUID="44d36a77-e403-4214-8e87-877e94cf4c79" containerName="revoke-keys" Oct 30 20:37:32.524363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.524360 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d36a77-e403-4214-8e87-877e94cf4c79" containerName="revoke-keys" Oct 30 20:37:32.524601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.524456 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d36a77-e403-4214-8e87-877e94cf4c79" containerName="revoke-keys" Oct 30 20:37:32.527942 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.527925 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.544674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.544651 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:37:32.544858 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.544721 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:37:32.544927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.544860 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:37:32.544927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.544892 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:37:32.546916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.546897 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4"] Oct 30 20:37:32.663892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.663865 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.664070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.663926 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqdp\" (UniqueName: \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.664070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.664012 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.765414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.765322 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.765414 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.765400 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqdp\" (UniqueName: \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.765690 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.765435 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.766133 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.766108 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.767936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.767912 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.783724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.783691 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqdp\" (UniqueName: \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\") pod \"maas-api-revoke-sub-3d4156f1-gdsg4\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:32.841392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:32.841365 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:33.141006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:33.140978 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4"] Oct 30 20:37:33.144860 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:37:33.144832 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15830e78_5cd5_4b7e_a38b_d6ceb51c9e95.slice/crio-f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9 WatchSource:0}: Error finding container f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9: Status 404 returned error can't find the container with id f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9 Oct 30 20:37:33.810553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:33.810456 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" event={"ID":"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95","Type":"ContainerStarted","Data":"dcd4726bd8163208b7011947d3c280c1be50d90185c80f9e9bb21e41f911ad06"} Oct 30 20:37:33.810553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:33.810539 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" event={"ID":"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95","Type":"ContainerStarted","Data":"f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9"} Oct 30 20:37:33.850718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:33.850664 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" podStartSLOduration=1.8506510569999999 podStartE2EDuration="1.850651057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:33.849655665 +0000 UTC m=+3777.978889586" watchObservedRunningTime="2026-10-30 20:37:33.850651057 +0000 UTC m=+3777.979885024" Oct 30 20:37:33.994936 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:33.994893 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21581fc-5382-4489-9774-82b9a9d4f330" path="/var/lib/kubelet/pods/c21581fc-5382-4489-9774-82b9a9d4f330/volumes" Oct 30 20:37:34.728536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:34.728463 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:34.816662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:34.816626 4959 generic.go:356] "Generic (PLEG): container finished" podID="15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" containerID="dcd4726bd8163208b7011947d3c280c1be50d90185c80f9e9bb21e41f911ad06" exitCode=0 Oct 30 20:37:34.817079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:34.816710 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" event={"ID":"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95","Type":"ContainerDied","Data":"dcd4726bd8163208b7011947d3c280c1be50d90185c80f9e9bb21e41f911ad06"} Oct 30 20:37:35.218345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:35.218315 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:35.971569 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:35.971546 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:36.098538 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.098444 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\" (UniqueName: \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\") pod \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " Oct 30 20:37:36.098696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.098587 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\") pod \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " Oct 30 20:37:36.098696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.098612 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\") pod \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\" (UID: \"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95\") " Oct 30 20:37:36.099055 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.099024 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca" pod "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" (UID: "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:37:36.100674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.100653 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp" pod "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" (UID: "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95"). InnerVolumeSpecName "kube-api-access-dpqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:37:36.100847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.100826 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca" pod "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" (UID: "15830e78-5cd5-4b7e-a38b-d6ceb51c9e95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:37:36.108289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.108266 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:36.200214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.200179 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpqdp\" (UniqueName: \"kubernetes.io/projected/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-kube-api-access-dpqdp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:36.200214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.200209 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:36.200214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.200220 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:37:36.829452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.829421 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" Oct 30 20:37:36.829648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.829448 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4" event={"ID":"15830e78-5cd5-4b7e-a38b-d6ceb51c9e95","Type":"ContainerDied","Data":"f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9"} Oct 30 20:37:36.829648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:36.829514 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86d42155cfe826712d83b20d7e7c518e7dd76d23bf6dcba3171c9eb311ff4a9" Oct 30 20:37:38.125074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:38.125045 4959 scope.go:122] "RemoveContainer" containerID="6b950922d56ecfecdfb75102ca09c16e519c25d15607b4aeacd114569c5a429f" Oct 30 20:37:38.134204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:38.134177 4959 scope.go:122] "RemoveContainer" containerID="33791713d5ebf7eab61a0a1f23725041e1962643d77c410ad67878e30d9552ab" Oct 30 20:37:46.216520 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:37:46.216358 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:37:53.965966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:53.965934 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.390175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.390140 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:37:54.390871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.390842 4959 cpu_manager.go:414] "containerMap: removing container" podUID="15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" containerName="revoke-keys" Oct 30 20:37:54.390871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.390868 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" containerName="revoke-keys" Oct 30 20:37:54.391122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.390989 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" containerName="revoke-keys" Oct 30 20:37:54.394341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.394315 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.413820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.413793 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-26e5b0-openshift-default-dockercfg-9sm4s\"" Oct 30 20:37:54.428967 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.428937 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:37:54.452004 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.451970 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.513855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.513814 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.513855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.513857 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.513996 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514123 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2k7\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514202 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514165 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514284 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514263 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514319 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514393 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514350 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.514452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.514396 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.615851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.615818 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.615886 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.615912 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.615956 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616005 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2k7\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616031 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616065 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616095 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616125 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616305 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616592 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616649 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616602 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.616714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616684 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.617003 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.616981 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.618671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.618649 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.618910 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.618892 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.619337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.619305 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:54.651268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.651177 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2k7\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.652141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.652089 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\") pod \"e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:54.708318 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:54.708285 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:55.026015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.025939 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:37:55.029526 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:37:55.029484 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdf69df_7c62_43f1_a909_94fe1a6416ce.slice/crio-b938697d6b4fb30f25fc816fe1d42a2709af5cdd9a42ca739b089c972d0a5c95 WatchSource:0}: Error finding container b938697d6b4fb30f25fc816fe1d42a2709af5cdd9a42ca739b089c972d0a5c95: Status 404 returned error can't find the container with id b938697d6b4fb30f25fc816fe1d42a2709af5cdd9a42ca739b089c972d0a5c95 Oct 30 20:37:55.031554 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.031519 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:37:55.031644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.031588 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:37:55.031644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.031620 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:37:55.968066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.968030 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerStarted","Data":"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a"} Oct 30 20:37:55.968268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:55.968092 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerStarted","Data":"b938697d6b4fb30f25fc816fe1d42a2709af5cdd9a42ca739b089c972d0a5c95"} Oct 30 20:37:56.028257 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:56.028202 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podStartSLOduration=2.028188685 podStartE2EDuration="2.028188685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:56.027997283 +0000 UTC m=+3800.157231204" watchObservedRunningTime="2026-10-30 20:37:56.028188685 +0000 UTC m=+3800.157422606" Oct 30 20:37:56.709192 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:56.709152 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:56.714459 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:56.714433 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:56.973026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:56.972951 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:56.973968 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:56.973946 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:37:58.508707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:58.508676 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:58.638015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:58.637984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.265390 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:59.265358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.370637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:59.370609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.963674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:37:59.963629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:00.075877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.075837 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:00.194551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.194518 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:00.302553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.302452 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:00.465662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.465596 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:38:00.471206 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.471172 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.481667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.481638 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:38:00.502445 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.502422 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4bf82de9-dockercfg-756zm\"" Oct 30 20:38:00.531470 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.531447 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-26e5b0\"" Oct 30 20:38:00.580362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.580303 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8vr\" (UniqueName: \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.580455 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.580446 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.686250 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.686208 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.686593 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.686563 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8vr\" (UniqueName: \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.688138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.688110 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.706056 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.706025 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8vr\" (UniqueName: \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\") pod \"payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:00.797179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:00.797147 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:01.102744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:01.102672 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:38:01.107113 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:38:01.107088 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd800c3_f5f2_45a7_a5f9_683fc8fa3892.slice/crio-e4eed59809b2aeea16e26594273bd4f4977c48659593b2c22c3a0904e41cd298 WatchSource:0}: Error finding container e4eed59809b2aeea16e26594273bd4f4977c48659593b2c22c3a0904e41cd298: Status 404 returned error can't find the container with id e4eed59809b2aeea16e26594273bd4f4977c48659593b2c22c3a0904e41cd298 Oct 30 20:38:01.593990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:01.593956 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:01.999383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:01.999348 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" event={"ID":"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892","Type":"ContainerStarted","Data":"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d"} Oct 30 20:38:01.999609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:01.999395 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" event={"ID":"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892","Type":"ContainerStarted","Data":"e4eed59809b2aeea16e26594273bd4f4977c48659593b2c22c3a0904e41cd298"} Oct 30 20:38:01.999609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:01.999426 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:02.053371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:02.053322 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" podStartSLOduration=2.053308778 podStartE2EDuration="2.053308778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:02.041746904 +0000 UTC m=+3806.170980825" watchObservedRunningTime="2026-10-30 20:38:02.053308778 +0000 UTC m=+3806.182542736" Oct 30 20:38:03.215006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:03.214966 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:05.012759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:05.012657 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:06.612594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:06.612561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:07.997743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:07.997681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:09.567197 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:09.567152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:10.986476 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:10.986388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.474104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:12.474066 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.600991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:12.599774 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": read tcp 10.132.0.2:41008->10.132.0.240:15021: read: connection reset by peer" start-of-body= Oct 30 20:38:12.600991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:12.599833 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": read tcp 10.132.0.2:41008->10.132.0.240:15021: read: connection reset by peer" Oct 30 20:38:13.008131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.008105 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:38:13.075720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.075682 4959 generic.go:356] "Generic (PLEG): container finished" podID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerID="bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a" exitCode=0 Oct 30 20:38:13.075951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.075785 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerDied","Data":"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a"} Oct 30 20:38:13.076204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.076186 4959 scope.go:122] "RemoveContainer" containerID="bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a" Oct 30 20:38:13.077338 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.077313 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:38:13.077399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.077378 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:38:13.077436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.077411 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:38:13.374672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.374563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:13.887152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:13.887108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.027348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.027313 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.083564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.083531 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerStarted","Data":"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd"} Oct 30 20:38:14.146479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.146383 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.277803 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.277772 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:14.709419 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.709386 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:38:14.709701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.709680 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 20:38:14.709759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:14.709740 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 20:38:15.587188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:15.587150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:15.709543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:15.709484 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 20:38:15.709722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:15.709594 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 20:38:15.878900 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:15.878807 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:16.103789 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:38:16.103753 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:38:16.709283 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:16.709207 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 20:38:16.709283 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:16.709256 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 20:38:17.237736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:17.237698 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:17.708745 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:17.708710 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 20:38:17.708972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:17.708767 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 20:38:18.709547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:18.709484 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 30 20:38:18.709960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:18.709563 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 30 20:38:19.085984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:19.085898 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:19.712288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:19.712253 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:38:19.712724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:19.712657 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:38:19.713228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:19.713211 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:38:20.593118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:20.593083 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:22.077820 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:22.077793 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:23.671001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:23.670956 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:25.168212 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:25.168177 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:26.687246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:26.687216 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:28.028551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:28.028519 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv"] Oct 30 20:38:28.039240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:28.039214 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtfgv"] Oct 30 20:38:28.221990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:28.221942 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:29.581542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:29.581515 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:29.994159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:29.994124 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3735022-3dec-4620-951d-b22d8af01861" path="/var/lib/kubelet/pods/a3735022-3dec-4620-951d-b22d8af01861/volumes" Oct 30 20:38:30.985768 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:30.985702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:32.449253 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:32.449216 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:33.989166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:33.989110 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:34.846424 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:34.846393 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:35.344180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:35.344147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:35.472960 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:35.472928 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:36.870679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:36.870648 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:38.159099 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:38.159067 4959 scope.go:122] "RemoveContainer" containerID="c679c41c1f82122f3a058da88a8242a7775338d4eaaf4b1214cbe5d0896a200d" Oct 30 20:38:38.272487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:38.272451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:39.670573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:39.670535 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:41.098073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:41.098043 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:42.583662 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:42.583613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:43.986779 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:43.986750 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:45.471892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.471847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:45.551104 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.551071 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w"] Oct 30 20:38:45.554810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.554787 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.574918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.574895 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w"] Oct 30 20:38:45.583703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.583634 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:38:45.583703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.583646 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:38:45.583703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.583697 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:38:45.583924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.583720 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:38:45.695115 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.695087 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.695254 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.695152 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.695310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.695280 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89sz\" (UniqueName: \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.796440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.796360 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.796440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.796413 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.796680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.796453 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89sz\" (UniqueName: \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.797060 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.797036 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.798997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.798973 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.815547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.815521 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89sz\" (UniqueName: \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\") pod \"maas-api-revoke-sub-8bfe6c0a-dj72w\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:45.868831 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:45.868806 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:46.065433 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:38:46.065354 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:38:46.169697 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:46.169669 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w"] Oct 30 20:38:46.173691 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:38:46.173664 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f9b3d4_1b90_4d87_80d4_9e8857a92cd4.slice/crio-07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2 WatchSource:0}: Error finding container 07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2: Status 404 returned error can't find the container with id 07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2 Oct 30 20:38:46.262069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:46.262034 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" event={"ID":"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4","Type":"ContainerStarted","Data":"07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2"} Oct 30 20:38:46.989170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:46.989133 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:47.043557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:47.043518 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84"] Oct 30 20:38:47.061481 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:47.061443 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-rzr84"] Oct 30 20:38:47.269330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:47.269237 4959 generic.go:356] "Generic (PLEG): container finished" podID="26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" containerID="a3926e3794e525a324cee9085940a4d30f0ff3460efa9c5d68e49624a6aa75f2" exitCode=0 Oct 30 20:38:47.269330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:47.269313 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" event={"ID":"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4","Type":"ContainerDied","Data":"a3926e3794e525a324cee9085940a4d30f0ff3460efa9c5d68e49624a6aa75f2"} Oct 30 20:38:47.995782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:47.995750 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb5315a8-33e4-4f93-b1b7-2685fb17e284" path="/var/lib/kubelet/pods/eb5315a8-33e4-4f93-b1b7-2685fb17e284/volumes" Oct 30 20:38:48.415418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.415391 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:48.522630 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.522603 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\") pod \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " Oct 30 20:38:48.522839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.522655 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\" (UniqueName: \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\") pod \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " Oct 30 20:38:48.522839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.522694 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\") pod \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\" (UID: \"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4\") " Oct 30 20:38:48.523095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.523072 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca" pod "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" (UID: "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:38:48.524995 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.524971 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca" pod "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" (UID: "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:38:48.525116 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.525064 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz" pod "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" (UID: "26f9b3d4-1b90-4d87-80d4-9e8857a92cd4"). InnerVolumeSpecName "kube-api-access-f89sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:48.624465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.624423 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:48.624465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.624457 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89sz\" (UniqueName: \"kubernetes.io/projected/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-kube-api-access-f89sz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:48.624465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.624467 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:48.898388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:48.898349 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:49.281736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:49.281656 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" Oct 30 20:38:49.282146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:49.281659 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w" event={"ID":"26f9b3d4-1b90-4d87-80d4-9e8857a92cd4","Type":"ContainerDied","Data":"07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2"} Oct 30 20:38:49.282146 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:49.281784 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b5f9b5b347cf406899655993d4da8c43fa4605de0dfe64169ebc6542faf7d2" Oct 30 20:38:50.384261 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:50.384226 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:51.861556 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:51.861487 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:53.272740 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:53.272710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:54.766894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:54.766850 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.288809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.288760 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.568150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.568064 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.783421 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.783381 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj"] Oct 30 20:38:56.783914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.783894 4959 cpu_manager.go:414] "containerMap: removing container" podUID="26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" containerName="revoke-keys" Oct 30 20:38:56.784019 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.783916 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" containerName="revoke-keys" Oct 30 20:38:56.784077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.784018 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" containerName="revoke-keys" Oct 30 20:38:56.786855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.786834 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.802326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.802302 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.802428 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.802418 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2lf\" (UniqueName: \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.802524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.802439 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.813551 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.813528 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:38:56.813675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.813620 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:38:56.813675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.813638 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:38:56.813819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.813749 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:38:56.817453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.817426 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj"] Oct 30 20:38:56.903553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.903486 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2lf\" (UniqueName: \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.903553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.903547 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.903853 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.903615 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.904294 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.904274 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.906022 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.906003 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:56.922507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:56.922465 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2lf\" (UniqueName: \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:57.100042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:57.100015 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:57.399658 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:57.399627 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj"] Oct 30 20:38:57.403939 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:38:57.403910 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39446e2_6203_44ce_ad04_885357d57677.slice/crio-689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470 WatchSource:0}: Error finding container 689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470: Status 404 returned error can't find the container with id 689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470 Oct 30 20:38:57.812398 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:57.812360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:58.334831 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:58.334795 4959 generic.go:356] "Generic (PLEG): container finished" podID="a39446e2-6203-44ce-ad04-885357d57677" containerID="228130f98e76156458e5ea3662f2465fc91ac35c2e9c13959e5268044ba92d1a" exitCode=0 Oct 30 20:38:58.334978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:58.334877 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" event={"ID":"a39446e2-6203-44ce-ad04-885357d57677","Type":"ContainerDied","Data":"228130f98e76156458e5ea3662f2465fc91ac35c2e9c13959e5268044ba92d1a"} Oct 30 20:38:58.334978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:58.334962 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" event={"ID":"a39446e2-6203-44ce-ad04-885357d57677","Type":"ContainerStarted","Data":"689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470"} Oct 30 20:38:59.030916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.030881 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8"] Oct 30 20:38:59.041600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.041574 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jxsp8"] Oct 30 20:38:59.298950 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.298872 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.496575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.496548 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:38:59.538512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.534816 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\") pod \"a39446e2-6203-44ce-ad04-885357d57677\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " Oct 30 20:38:59.538512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.534917 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\") pod \"a39446e2-6203-44ce-ad04-885357d57677\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " Oct 30 20:38:59.538512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.535004 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\" (UniqueName: \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\") pod \"a39446e2-6203-44ce-ad04-885357d57677\" (UID: \"a39446e2-6203-44ce-ad04-885357d57677\") " Oct 30 20:38:59.538512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.536701 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca" pod "a39446e2-6203-44ce-ad04-885357d57677" (UID: "a39446e2-6203-44ce-ad04-885357d57677"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:38:59.541140 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.541104 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca" pod "a39446e2-6203-44ce-ad04-885357d57677" (UID: "a39446e2-6203-44ce-ad04-885357d57677"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:38:59.608558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.607337 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf" pod "a39446e2-6203-44ce-ad04-885357d57677" (UID: "a39446e2-6203-44ce-ad04-885357d57677"). InnerVolumeSpecName "kube-api-access-wb2lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:38:59.643521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.636090 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb2lf\" (UniqueName: \"kubernetes.io/projected/a39446e2-6203-44ce-ad04-885357d57677-kube-api-access-wb2lf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:59.643521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.636122 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39446e2-6203-44ce-ad04-885357d57677-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:59.643521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.636138 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39446e2-6203-44ce-ad04-885357d57677-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:38:59.995184 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:38:59.995149 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a16d503-2d9d-4314-acd3-ddcfddb7a837" path="/var/lib/kubelet/pods/0a16d503-2d9d-4314-acd3-ddcfddb7a837/volumes" Oct 30 20:39:00.354385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:00.354303 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" event={"ID":"a39446e2-6203-44ce-ad04-885357d57677","Type":"ContainerDied","Data":"689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470"} Oct 30 20:39:00.354385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:00.354319 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj" Oct 30 20:39:00.354385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:00.354351 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="689ca27ded02042fb735b9c13a75a28f877dec5dd6eef7b4d3ef0dff56abc470" Oct 30 20:39:00.861603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:00.861568 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.602687 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:01.602656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.384543 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:02.384510 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.989955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:02.989923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.096997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:03.096963 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.884272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:03.884241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:05.264599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:05.264567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:06.671581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:06.671547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:06.887262 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:06.887227 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:39:07.023879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:07.023783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:07.135992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:07.135917 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:08.171701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:08.171666 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:09.455227 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:09.455177 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:09.882819 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:09.882787 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:09.984925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:09.984896 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj"] Oct 30 20:39:09.993720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:09.993681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:09.995460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:09.995422 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-26e5b0-tv6fj"] Oct 30 20:39:10.540691 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:10.540659 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:11.890670 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:11.890601 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" containerID="cri-o://681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d" gracePeriod=25 Oct 30 20:39:11.994961 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:11.994929 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39446e2-6203-44ce-ad04-885357d57677" path="/var/lib/kubelet/pods/a39446e2-6203-44ce-ad04-885357d57677/volumes" Oct 30 20:39:13.006047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:13.006011 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:39:16.065106 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:39:16.065069 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:39:16.420810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:16.420773 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:16.581296 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:16.581263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:16.740362 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:16.740268 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:39:16.740583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:16.740537 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" containerID="cri-o://5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd" gracePeriod=30 Oct 30 20:39:16.796837 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:16.796808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:18.004894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.004856 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4"] Oct 30 20:39:18.016289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.016254 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gdsg4"] Oct 30 20:39:18.058030 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.057989 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm"] Oct 30 20:39:18.058949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.058922 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a39446e2-6203-44ce-ad04-885357d57677" containerName="revoke-keys" Oct 30 20:39:18.058949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.058949 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39446e2-6203-44ce-ad04-885357d57677" containerName="revoke-keys" Oct 30 20:39:18.059148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.059130 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39446e2-6203-44ce-ad04-885357d57677" containerName="revoke-keys" Oct 30 20:39:18.064344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.064319 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.076376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.076353 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm"] Oct 30 20:39:18.078010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.077985 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:39:18.078152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.078031 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:39:18.078152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.078048 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:39:18.078152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.078133 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:39:18.227835 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.227795 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.228027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.227949 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.228027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.228013 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8t6\" (UniqueName: \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.328773 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.328689 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.328947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.328776 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.328947 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.328800 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8t6\" (UniqueName: \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.329383 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.329358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.331301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.331273 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.347106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.347078 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8t6\" (UniqueName: \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\") pod \"maas-api-revoke-sub-3d4156f1-m4sdm\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.381074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.381032 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:18.679824 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:18.679787 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm"] Oct 30 20:39:18.684527 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:39:18.684471 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0814b3e6_c01e_4b95_b81a_f617634a879d.slice/crio-21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8 WatchSource:0}: Error finding container 21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8: Status 404 returned error can't find the container with id 21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8 Oct 30 20:39:19.473722 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.473683 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" event={"ID":"0814b3e6-c01e-4b95-b81a-f617634a879d","Type":"ContainerStarted","Data":"f4e074c08fa6d265a39fda0d461f873f5ba95805131f10273f98d92b4eb0a4dc"} Oct 30 20:39:19.474121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.473751 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" event={"ID":"0814b3e6-c01e-4b95-b81a-f617634a879d","Type":"ContainerStarted","Data":"21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8"} Oct 30 20:39:19.511252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.511204 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" podStartSLOduration=1.5111904520000001 podStartE2EDuration="1.511190452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:19.510025871 +0000 UTC m=+3883.639259791" watchObservedRunningTime="2026-10-30 20:39:19.511190452 +0000 UTC m=+3883.640424374" Oct 30 20:39:19.715286 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.715243 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:39:19.715442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.715298 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:39:19.996023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:19.995985 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15830e78-5cd5-4b7e-a38b-d6ceb51c9e95" path="/var/lib/kubelet/pods/15830e78-5cd5-4b7e-a38b-d6ceb51c9e95/volumes" Oct 30 20:39:20.479798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:20.479764 4959 generic.go:356] "Generic (PLEG): container finished" podID="0814b3e6-c01e-4b95-b81a-f617634a879d" containerID="f4e074c08fa6d265a39fda0d461f873f5ba95805131f10273f98d92b4eb0a4dc" exitCode=0 Oct 30 20:39:20.480248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:20.479850 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" event={"ID":"0814b3e6-c01e-4b95-b81a-f617634a879d","Type":"ContainerDied","Data":"f4e074c08fa6d265a39fda0d461f873f5ba95805131f10273f98d92b4eb0a4dc"} Oct 30 20:39:21.623694 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.623665 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:21.762344 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.762320 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\") pod \"0814b3e6-c01e-4b95-b81a-f617634a879d\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " Oct 30 20:39:21.762486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.762444 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\") pod \"0814b3e6-c01e-4b95-b81a-f617634a879d\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " Oct 30 20:39:21.762486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.762467 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\" (UniqueName: \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\") pod \"0814b3e6-c01e-4b95-b81a-f617634a879d\" (UID: \"0814b3e6-c01e-4b95-b81a-f617634a879d\") " Oct 30 20:39:21.762855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.762832 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca" pod "0814b3e6-c01e-4b95-b81a-f617634a879d" (UID: "0814b3e6-c01e-4b95-b81a-f617634a879d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:21.764667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.764641 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca" pod "0814b3e6-c01e-4b95-b81a-f617634a879d" (UID: "0814b3e6-c01e-4b95-b81a-f617634a879d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:39:21.764770 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.764726 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6" pod "0814b3e6-c01e-4b95-b81a-f617634a879d" (UID: "0814b3e6-c01e-4b95-b81a-f617634a879d"). InnerVolumeSpecName "kube-api-access-5k8t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:21.863646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.863622 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:21.863646 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.863648 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8t6\" (UniqueName: \"kubernetes.io/projected/0814b3e6-c01e-4b95-b81a-f617634a879d-kube-api-access-5k8t6\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:21.863802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.863666 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0814b3e6-c01e-4b95-b81a-f617634a879d-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:21.991913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:21.991889 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:39:22.166677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166652 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.166828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166692 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.166828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166735 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.166828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166783 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.166986 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166966 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:22.167040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.166982 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.167040 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167031 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.167142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167066 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.167204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167157 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.167260 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167215 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\") pod \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\" (UID: \"2cdf69df-7c62-43f1-a909-94fe1a6416ce\") " Oct 30 20:39:22.167750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167404 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:22.167750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167546 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:22.167750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167644 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:22.167750 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167439 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:22.168249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167774 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.168249 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.167802 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.169087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.169060 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:39:22.169246 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.169226 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:22.169324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.169257 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "kube-api-access-5q2k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:22.169482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.169468 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo" pod "2cdf69df-7c62-43f1-a909-94fe1a6416ce" (UID: "2cdf69df-7c62-43f1-a909-94fe1a6416ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:39:22.268975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.268954 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.268975 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.268975 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.269108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.268983 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.269108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.268993 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2k7\" (UniqueName: \"kubernetes.io/projected/2cdf69df-7c62-43f1-a909-94fe1a6416ce-kube-api-access-5q2k7\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.269108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.269001 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdf69df-7c62-43f1-a909-94fe1a6416ce-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.269108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.269009 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.269108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.269016 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdf69df-7c62-43f1-a909-94fe1a6416ce-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:22.491632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.491545 4959 generic.go:356] "Generic (PLEG): container finished" podID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerID="5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd" exitCode=0 Oct 30 20:39:22.491632 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.491601 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" Oct 30 20:39:22.491904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.491642 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerDied","Data":"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd"} Oct 30 20:39:22.491904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.491715 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz" event={"ID":"2cdf69df-7c62-43f1-a909-94fe1a6416ce","Type":"ContainerDied","Data":"b938697d6b4fb30f25fc816fe1d42a2709af5cdd9a42ca739b089c972d0a5c95"} Oct 30 20:39:22.491904 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.491745 4959 scope.go:122] "RemoveContainer" containerID="5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd" Oct 30 20:39:22.493136 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.493105 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" event={"ID":"0814b3e6-c01e-4b95-b81a-f617634a879d","Type":"ContainerDied","Data":"21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8"} Oct 30 20:39:22.493195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.493140 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21698074d8600b38ab286c49cb96ff215290cdff4d6c446a14e19229cdb3fca8" Oct 30 20:39:22.493195 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.493111 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm" Oct 30 20:39:22.501683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.501665 4959 scope.go:122] "RemoveContainer" containerID="bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a" Oct 30 20:39:22.509928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.509910 4959 scope.go:122] "RemoveContainer" containerID="5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd" Oct 30 20:39:22.510201 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:39:22.510184 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd\": container with ID starting with 5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd not found: ID does not exist" containerID="5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd" Oct 30 20:39:22.510326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.510209 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd"} err="failed to get container status \"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd\": rpc error: code = NotFound desc = could not find container \"5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd\": container with ID starting with 5d4b0d05fa22ac256d79254363c47d2040fa8482c0b6407b1c2ad8706c348edd not found: ID does not exist" Oct 30 20:39:22.510326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.510226 4959 scope.go:122] "RemoveContainer" containerID="bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a" Oct 30 20:39:22.510478 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:39:22.510463 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a\": container with ID starting with bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a not found: ID does not exist" containerID="bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a" Oct 30 20:39:22.510623 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.510481 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a"} err="failed to get container status \"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a\": rpc error: code = NotFound desc = could not find container \"bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a\": container with ID starting with bea406a908347418534f60a8537308f5f2b02085dbcaf80a9533814a3e35051a not found: ID does not exist" Oct 30 20:39:22.539802 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.539780 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:39:22.551084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:22.551065 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26e5b0-openshift-default-6cd97fbcd5c2qnz"] Oct 30 20:39:23.005637 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:23.005598 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:39:23.994218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:23.994187 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" path="/var/lib/kubelet/pods/2cdf69df-7c62-43f1-a909-94fe1a6416ce/volumes" Oct 30 20:39:26.035348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:26.035318 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2"] Oct 30 20:39:26.045977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:26.045953 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-llsg2"] Oct 30 20:39:27.994719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:27.994687 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e4f6f8-69a3-4167-855e-ecc41543847e" path="/var/lib/kubelet/pods/e9e4f6f8-69a3-4167-855e-ecc41543847e/volumes" Oct 30 20:39:32.448038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.448005 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:39:32.518547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.518519 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\" (UniqueName: \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\") pod \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " Oct 30 20:39:32.518739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.518567 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\") pod \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\" (UID: \"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892\") " Oct 30 20:39:32.518991 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.518965 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume" pod "8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" (UID: "8cd800c3-f5f2-45a7-a5f9-683fc8fa3892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:39:32.520914 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.520894 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr" pod "8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" (UID: "8cd800c3-f5f2-45a7-a5f9-683fc8fa3892"). InnerVolumeSpecName "kube-api-access-7n8vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:39:32.597094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.597011 4959 generic.go:356] "Generic (PLEG): container finished" podID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerID="681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d" exitCode=0 Oct 30 20:39:32.597094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.597053 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" event={"ID":"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892","Type":"ContainerDied","Data":"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d"} Oct 30 20:39:32.597094 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.597060 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" Oct 30 20:39:32.597353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.597103 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx" event={"ID":"8cd800c3-f5f2-45a7-a5f9-683fc8fa3892","Type":"ContainerDied","Data":"e4eed59809b2aeea16e26594273bd4f4977c48659593b2c22c3a0904e41cd298"} Oct 30 20:39:32.597353 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.597126 4959 scope.go:122] "RemoveContainer" containerID="681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d" Oct 30 20:39:32.606981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.606952 4959 scope.go:122] "RemoveContainer" containerID="681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d" Oct 30 20:39:32.607270 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:39:32.607247 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d\": container with ID starting with 681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d not found: ID does not exist" containerID="681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d" Oct 30 20:39:32.607329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.607279 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d"} err="failed to get container status \"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d\": rpc error: code = NotFound desc = could not find container \"681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d\": container with ID starting with 681653ac9a48b00f796c7df571bb1119f2826b7be07d8341039aced0a175651d not found: ID does not exist" Oct 30 20:39:32.619154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.619131 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8vr\" (UniqueName: \"kubernetes.io/projected/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-kube-api-access-7n8vr\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:32.619154 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.619151 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:39:32.645956 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.645933 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:39:32.652581 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:32.652559 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26e5b0-7f97f666djxhwx"] Oct 30 20:39:33.994337 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:33.994307 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" path="/var/lib/kubelet/pods/8cd800c3-f5f2-45a7-a5f9-683fc8fa3892/volumes" Oct 30 20:39:34.969600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:34.969572 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:35.504156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:35.504124 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:36.805139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:36.805114 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:39:36.805680 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:36.805663 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:39:37.027044 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:37.027016 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6"] Oct 30 20:39:37.037901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:37.037879 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tf2m6"] Oct 30 20:39:37.994166 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:37.994130 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c6892e-2034-4a40-8037-bfe2ee16d1c6" path="/var/lib/kubelet/pods/e0c6892e-2034-4a40-8037-bfe2ee16d1c6/volumes" Oct 30 20:39:38.180349 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:38.180316 4959 scope.go:122] "RemoveContainer" containerID="bfada8d33964198cbb0a1694115a94c948b8091bf19872331bff276dc1abf184" Oct 30 20:39:38.293093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:38.292999 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:39.165959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:39.165925 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:45.366955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:45.366924 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:46.065028 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:39:46.064996 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:39:58.025323 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:58.025289 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l"] Oct 30 20:39:58.036129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:58.036106 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d594l"] Oct 30 20:39:59.994266 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:39:59.994221 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc030a2-7c59-41f9-855e-93638b6d6e2f" path="/var/lib/kubelet/pods/0cc030a2-7c59-41f9-855e-93638b6d6e2f/volumes" Oct 30 20:40:16.065744 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:40:16.065707 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:40:20.027952 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:20.027919 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2"] Oct 30 20:40:20.038541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:20.038513 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-z7qw2"] Oct 30 20:40:21.994223 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:21.994189 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e808dfe-e073-43c1-92f0-30885d1e7f0d" path="/var/lib/kubelet/pods/1e808dfe-e073-43c1-92f0-30885d1e7f0d/volumes" Oct 30 20:40:35.145595 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:35.145536 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:35.577469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:35.577370 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:38.212887 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:38.212852 4959 scope.go:122] "RemoveContainer" containerID="5a78e762d56d99ccbc0e15fb5a05ee54fa21a51017082ad814ab06eac58daa60" Oct 30 20:40:38.221735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:38.221717 4959 scope.go:122] "RemoveContainer" containerID="315804a1fcf03fa80a5257fed8b5cc47edbef09778b56cafc97b97a6974229ff" Oct 30 20:40:38.232071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:38.232051 4959 scope.go:122] "RemoveContainer" containerID="1c1371bd06242fab381e9ab70d5ed6b194b5889e144009cec6ade2a5de832de8" Oct 30 20:40:38.239550 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:38.239529 4959 scope.go:122] "RemoveContainer" containerID="df75513d31fef1bf17d5c0d4622acf8ece956f9ed82b7f4c8ece8e4eccd41a7d" Oct 30 20:40:41.026215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:41.026184 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn"] Oct 30 20:40:41.037119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:41.037096 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dc2rn"] Oct 30 20:40:41.995865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:40:41.995833 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7832e75-938b-49e9-810e-17d8816e42dd" path="/var/lib/kubelet/pods/f7832e75-938b-49e9-810e-17d8816e42dd/volumes" Oct 30 20:40:46.064850 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:40:46.064816 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:41:01.029896 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:01.029858 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq"] Oct 30 20:41:01.040767 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:01.040740 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9kxrq"] Oct 30 20:41:01.994641 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:01.994608 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da" path="/var/lib/kubelet/pods/56c8da5a-9c30-4fb7-9e7e-94c3cc3cc5da/volumes" Oct 30 20:41:07.167084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167049 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9"] Oct 30 20:41:07.167679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167627 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0814b3e6-c01e-4b95-b81a-f617634a879d" containerName="revoke-keys" Oct 30 20:41:07.167679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167645 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0814b3e6-c01e-4b95-b81a-f617634a879d" containerName="revoke-keys" Oct 30 20:41:07.167679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167673 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167679 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167678 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167688 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167693 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167702 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167706 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167793 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167801 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdf69df-7c62-43f1-a909-94fe1a6416ce" containerName="istio-proxy" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167809 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd800c3-f5f2-45a7-a5f9-683fc8fa3892" containerName="payload-processing" Oct 30 20:41:07.167834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.167817 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0814b3e6-c01e-4b95-b81a-f617634a879d" containerName="revoke-keys" Oct 30 20:41:07.171933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.171916 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.192041 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.192020 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9"] Oct 30 20:41:07.200732 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.200709 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:41:07.200869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.200737 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:41:07.200869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.200755 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:41:07.200869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.200739 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:41:07.255600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.255579 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.255701 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.255619 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.255752 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.255707 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.356958 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.356930 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.357087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.356984 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.357087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.357049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.357681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.357660 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.359480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.359460 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.376028 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.376005 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\") pod \"maas-api-revoke-sub-d576e68b-kd5d9\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.484545 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.484459 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:07.782208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:07.782121 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9"] Oct 30 20:41:07.786777 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:41:07.786747 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e739245_6510_4d14_9bb1_472a25715f11.slice/crio-f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67 WatchSource:0}: Error finding container f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67: Status 404 returned error can't find the container with id f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67 Oct 30 20:41:08.094420 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:08.094391 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" event={"ID":"9e739245-6510-4d14-9bb1-472a25715f11","Type":"ContainerStarted","Data":"19a66b047d8a18f241a603a6a72eb5ad95570667acb232b6933aa313531640e4"} Oct 30 20:41:08.094695 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:08.094439 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" event={"ID":"9e739245-6510-4d14-9bb1-472a25715f11","Type":"ContainerStarted","Data":"f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67"} Oct 30 20:41:08.135744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:08.135691 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" podStartSLOduration=1.13567678 podStartE2EDuration="1.13567678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:08.134320804 +0000 UTC m=+3992.263554725" watchObservedRunningTime="2026-10-30 20:41:08.13567678 +0000 UTC m=+3992.264910702" Oct 30 20:41:09.100859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:09.100762 4959 generic.go:356] "Generic (PLEG): container finished" podID="9e739245-6510-4d14-9bb1-472a25715f11" containerID="19a66b047d8a18f241a603a6a72eb5ad95570667acb232b6933aa313531640e4" exitCode=0 Oct 30 20:41:09.101221 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:09.100849 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" event={"ID":"9e739245-6510-4d14-9bb1-472a25715f11","Type":"ContainerDied","Data":"19a66b047d8a18f241a603a6a72eb5ad95570667acb232b6933aa313531640e4"} Oct 30 20:41:09.813571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:09.813541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:10.240278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.240254 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:10.386966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.386941 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\") pod \"9e739245-6510-4d14-9bb1-472a25715f11\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " Oct 30 20:41:10.387106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.387004 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\") pod \"9e739245-6510-4d14-9bb1-472a25715f11\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " Oct 30 20:41:10.387106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.387075 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\") pod \"9e739245-6510-4d14-9bb1-472a25715f11\" (UID: \"9e739245-6510-4d14-9bb1-472a25715f11\") " Oct 30 20:41:10.387363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.387342 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca" pod "9e739245-6510-4d14-9bb1-472a25715f11" (UID: "9e739245-6510-4d14-9bb1-472a25715f11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:10.389272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.389244 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca" pod "9e739245-6510-4d14-9bb1-472a25715f11" (UID: "9e739245-6510-4d14-9bb1-472a25715f11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:10.389391 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.389274 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc" pod "9e739245-6510-4d14-9bb1-472a25715f11" (UID: "9e739245-6510-4d14-9bb1-472a25715f11"). InnerVolumeSpecName "kube-api-access-8csjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:10.488119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.488074 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:41:10.488119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.488114 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e739245-6510-4d14-9bb1-472a25715f11-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:41:10.488307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.488130 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8csjc\" (UniqueName: \"kubernetes.io/projected/9e739245-6510-4d14-9bb1-472a25715f11-kube-api-access-8csjc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:41:10.563759 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:10.563726 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:11.111090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:11.111057 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" event={"ID":"9e739245-6510-4d14-9bb1-472a25715f11","Type":"ContainerDied","Data":"f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67"} Oct 30 20:41:11.111090 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:11.111074 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9" Oct 30 20:41:11.111314 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:11.111107 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13550ac384d67db701b24b5d457604f4b553592ebe222e5ff981d6874a35e67" Oct 30 20:41:16.067132 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:41:16.067098 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:41:35.203541 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:35.203488 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:38.269669 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:38.269639 4959 scope.go:122] "RemoveContainer" containerID="6f2732686acf728eb1e113b8c01508509c87ae338d67fe54c676e56b9d326c7f" Oct 30 20:41:38.278629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:38.278610 4959 scope.go:122] "RemoveContainer" containerID="40a9956930b9ab904a3d7929998ae83905076d84215987a14fb4bb548d531699" Oct 30 20:41:38.286404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:41:38.286378 4959 scope.go:122] "RemoveContainer" containerID="3712d8e66d3b0f8a7c223be5abc8d5da2989e32505e8aa604103a76a38eecad9" Oct 30 20:41:46.064833 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:41:46.064784 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:42:06.033913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:06.033883 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s"] Oct 30 20:42:06.045150 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:06.045125 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-fzk8s"] Oct 30 20:42:07.994347 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:07.994310 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b" path="/var/lib/kubelet/pods/b5d6ee3a-f460-4ceb-a3e8-a15584b9bf1b/volumes" Oct 30 20:42:14.215002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:14.214957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:15.126169 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:15.126140 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:16.064652 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:42:16.064610 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:42:17.043180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:17.043141 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj"] Oct 30 20:42:17.052446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:17.052415 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-qk4lj"] Oct 30 20:42:17.994401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:17.994367 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d36a77-e403-4214-8e87-877e94cf4c79" path="/var/lib/kubelet/pods/44d36a77-e403-4214-8e87-877e94cf4c79/volumes" Oct 30 20:42:20.937043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.936988 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4"] Oct 30 20:42:20.937686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.937456 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9e739245-6510-4d14-9bb1-472a25715f11" containerName="revoke-keys" Oct 30 20:42:20.937686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.937474 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e739245-6510-4d14-9bb1-472a25715f11" containerName="revoke-keys" Oct 30 20:42:20.937686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.937623 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e739245-6510-4d14-9bb1-472a25715f11" containerName="revoke-keys" Oct 30 20:42:20.941012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.940990 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:20.954901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.954881 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:42:20.957696 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.957683 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:42:20.966279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.966258 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:42:20.968002 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.967979 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4"] Oct 30 20:42:20.969119 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:20.969100 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:42:21.044884 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.044857 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.045020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.044886 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpvc\" (UniqueName: \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.045020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.044932 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.145486 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.145455 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.145664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.145568 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.145664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.145601 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpvc\" (UniqueName: \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.146292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.146226 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.148038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.148015 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.164793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.164766 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpvc\" (UniqueName: \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\") pod \"maas-api-revoke-sub-d2bc93bc-bksx4\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.253141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.253083 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:21.550908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:21.550831 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4"] Oct 30 20:42:21.555710 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:42:21.555673 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58862520_9f9c_4721_b908_8e30f67df475.slice/crio-82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060 WatchSource:0}: Error finding container 82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060: Status 404 returned error can't find the container with id 82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060 Oct 30 20:42:22.477288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:22.477257 4959 generic.go:356] "Generic (PLEG): container finished" podID="58862520-9f9c-4721-b908-8e30f67df475" containerID="39d061cef6baa36b76bdb20b86fcf280ace0e9fb92d7321a07e66bfe1e5f6f80" exitCode=0 Oct 30 20:42:22.477677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:22.477342 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" event={"ID":"58862520-9f9c-4721-b908-8e30f67df475","Type":"ContainerDied","Data":"39d061cef6baa36b76bdb20b86fcf280ace0e9fb92d7321a07e66bfe1e5f6f80"} Oct 30 20:42:22.477677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:22.477412 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" event={"ID":"58862520-9f9c-4721-b908-8e30f67df475","Type":"ContainerStarted","Data":"82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060"} Oct 30 20:42:23.620725 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.620698 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:23.669656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.669632 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\") pod \"58862520-9f9c-4721-b908-8e30f67df475\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " Oct 30 20:42:23.669774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.669690 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\" (UniqueName: \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\") pod \"58862520-9f9c-4721-b908-8e30f67df475\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " Oct 30 20:42:23.669774 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.669712 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\") pod \"58862520-9f9c-4721-b908-8e30f67df475\" (UID: \"58862520-9f9c-4721-b908-8e30f67df475\") " Oct 30 20:42:23.669981 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.669958 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca" pod "58862520-9f9c-4721-b908-8e30f67df475" (UID: "58862520-9f9c-4721-b908-8e30f67df475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:23.671930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.671909 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca" pod "58862520-9f9c-4721-b908-8e30f67df475" (UID: "58862520-9f9c-4721-b908-8e30f67df475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:23.671930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.671914 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc" pod "58862520-9f9c-4721-b908-8e30f67df475" (UID: "58862520-9f9c-4721-b908-8e30f67df475"). InnerVolumeSpecName "kube-api-access-llpvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:23.770537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.770430 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58862520-9f9c-4721-b908-8e30f67df475-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:42:23.770537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.770459 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llpvc\" (UniqueName: \"kubernetes.io/projected/58862520-9f9c-4721-b908-8e30f67df475-kube-api-access-llpvc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:42:23.770537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:23.770474 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58862520-9f9c-4721-b908-8e30f67df475-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:42:24.488809 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:24.488779 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" Oct 30 20:42:24.488984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:24.488823 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4" event={"ID":"58862520-9f9c-4721-b908-8e30f67df475","Type":"ContainerDied","Data":"82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060"} Oct 30 20:42:24.488984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:24.488869 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82b9cb76a8c436967db13a79dbe004af13dd94501daf870a2ebb7e1602096060" Oct 30 20:42:32.824079 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:32.824042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:33.242523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.239571 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:42:33.243368 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.243340 4959 cpu_manager.go:414] "containerMap: removing container" podUID="58862520-9f9c-4721-b908-8e30f67df475" containerName="revoke-keys" Oct 30 20:42:33.243549 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.243534 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58862520-9f9c-4721-b908-8e30f67df475" containerName="revoke-keys" Oct 30 20:42:33.243829 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.243810 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="58862520-9f9c-4721-b908-8e30f67df475" containerName="revoke-keys" Oct 30 20:42:33.330863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.330819 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:42:33.331270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.331248 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.387870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.387839 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388020 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388107 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388166 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388229 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388275 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388346 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388376 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388544 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsn9m\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.388760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.388585 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.410518 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.400777 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-051802-openshift-default-dockercfg-ccnn5\"" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501746 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsn9m\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501818 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501887 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501921 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501950 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.501989 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.502017 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.502093 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.502123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.502557 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.503166 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.503420 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.503694 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.509566 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.504329 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.542542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.536931 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.636519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.584119 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.636519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.602048 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.636519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.624402 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:33.636519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.630517 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsn9m\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\") pod \"e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:33.782571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:33.779726 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:34.539831 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:34.539796 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:42:34.610880 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:34.610842 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:34.610997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:34.610916 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:34.610997 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:34.610954 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:34.682437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:34.682402 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerStarted","Data":"eee5081127424c173c17fcc0a63ddbfc3d32380204bd498a80065cbed8f1dd43"} Oct 30 20:42:35.475699 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:35.475659 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:35.689611 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:35.689576 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerStarted","Data":"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8"} Oct 30 20:42:35.753723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:35.753638 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podStartSLOduration=2.753624817 podStartE2EDuration="2.753624817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:35.741417165 +0000 UTC m=+4079.870651118" watchObservedRunningTime="2026-10-30 20:42:35.753624817 +0000 UTC m=+4079.882858738" Oct 30 20:42:35.780796 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:35.780766 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:35.785642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:35.785620 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:36.701963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:36.701929 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:36.702931 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:36.702911 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:38.151963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:38.151923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:38.263776 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:38.263743 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:38.311741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:38.311713 4959 scope.go:122] "RemoveContainer" containerID="67fc4908ec1ef231300b7139c4dc4b275f589adbf539daa9a3f2de5b4f6b8e16" Oct 30 20:42:38.320392 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:38.320376 4959 scope.go:122] "RemoveContainer" containerID="fc5986af3ca8d92d098b5616edefbc39a6a53207d9a42d49b56e2b065bfd2bc1" Oct 30 20:42:38.328410 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:38.328393 4959 scope.go:122] "RemoveContainer" containerID="89c91b320e0489fc80d2633aa531821090a08f9b918c36c227a9839c4edc1e70" Oct 30 20:42:39.606263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:39.606232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:39.722648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:39.722616 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:39.851217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:39.851183 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:40.012976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.012945 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:40.041205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.041177 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:42:40.054388 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.054368 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.067190 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.067157 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:42:40.077405 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.077378 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-051802\"" Oct 30 20:42:40.077601 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.077441 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5707a02d-dockercfg-24g6d\"" Oct 30 20:42:40.236713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.236676 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.236869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.236735 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2xc\" (UniqueName: \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.337563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.337463 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.337742 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.337565 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2xc\" (UniqueName: \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.338207 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.338189 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.357075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.357047 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2xc\" (UniqueName: \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\") pod \"payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.366793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.366774 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:40.667077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.667050 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:42:40.671857 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:42:40.671827 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2d8597_deab_4050_9f26_b8b7e69aea67.slice/crio-6bfb0bdb4510007e10cb26cd0acfa78d064bdbd9354e27717b1453e3a1b3aac7 WatchSource:0}: Error finding container 6bfb0bdb4510007e10cb26cd0acfa78d064bdbd9354e27717b1453e3a1b3aac7: Status 404 returned error can't find the container with id 6bfb0bdb4510007e10cb26cd0acfa78d064bdbd9354e27717b1453e3a1b3aac7 Oct 30 20:42:40.726561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:40.726532 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" event={"ID":"ae2d8597-deab-4050-9f26-b8b7e69aea67","Type":"ContainerStarted","Data":"6bfb0bdb4510007e10cb26cd0acfa78d064bdbd9354e27717b1453e3a1b3aac7"} Oct 30 20:42:41.195403 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:41.195372 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:41.732798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:41.732762 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" event={"ID":"ae2d8597-deab-4050-9f26-b8b7e69aea67","Type":"ContainerStarted","Data":"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38"} Oct 30 20:42:41.733201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:41.732887 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:41.777297 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:41.777241 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" podStartSLOduration=1.777228741 podStartE2EDuration="1.777228741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:41.775869512 +0000 UTC m=+4085.905103427" watchObservedRunningTime="2026-10-30 20:42:41.777228741 +0000 UTC m=+4085.906462658" Oct 30 20:42:42.686198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:42.686169 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:44.273946 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:44.273917 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:45.794001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:45.793965 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:46.084824 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:42:46.084367 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:42:47.397299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:47.397271 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:47.904674 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:47.904645 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:48.583709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:48.583674 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:48.748479 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:48.748419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:48.907800 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:48.907768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:49.326523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:49.326408 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:50.500762 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:50.500727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:50.992033 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:50.991996 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:52.519312 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.519272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:52.745376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.744662 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:42:52.745376 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.744727 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:42:52.754193 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.754167 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:42:52.822665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.822570 4959 generic.go:356] "Generic (PLEG): container finished" podID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerID="5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8" exitCode=0 Oct 30 20:42:52.822842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.822699 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerDied","Data":"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8"} Oct 30 20:42:52.823172 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.823149 4959 scope.go:122] "RemoveContainer" containerID="5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8" Oct 30 20:42:52.824888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.824851 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:52.825014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.824942 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:52.825014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:52.824985 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:42:53.829503 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:53.829458 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerStarted","Data":"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea"} Oct 30 20:42:54.087287 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:54.087206 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:54.780901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:54.780867 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:54.781091 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:54.781073 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 20:42:54.781152 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:54.781126 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 20:42:55.489769 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:55.489733 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:55.781059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:55.780969 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 20:42:55.781059 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:55.781027 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 20:42:56.780714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:56.780683 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 20:42:56.781130 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:56.780753 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 20:42:56.979821 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:56.979791 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:57.780789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:57.780754 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 20:42:57.781196 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:57.780812 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 20:42:58.481645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:58.481610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:58.780944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:58.780846 4959 patch_prober.go:29] interesting pod/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 20:42:58.780944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:58.780896 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 20:42:59.784345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:59.784315 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:59.784788 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:59.784578 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:59.785320 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:59.785300 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:42:59.996466 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:42:59.996431 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:01.579881 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:01.579846 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:03.133134 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:03.132984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:04.502175 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:04.502144 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:05.986024 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:05.985984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:07.605980 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:07.605947 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:08.904252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:08.904213 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:10.455329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:10.455290 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:11.993006 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:11.992974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:13.686252 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:13.686204 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:15.356355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:15.356305 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:16.064104 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:43:16.064071 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:43:16.903438 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:16.903399 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:18.601689 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:18.601656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:20.177507 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:20.177467 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:21.771117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:21.771086 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:23.278275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:23.278240 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:24.698645 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:24.698598 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:26.201808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:26.201780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:27.925983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:27.925944 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:29.524395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:29.524359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:31.079926 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:31.079891 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:32.564642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:32.564613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:34.067243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:34.067211 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:35.443018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:35.442988 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:35.617274 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:35.617238 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:37.160069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:37.160028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:38.354436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:38.354404 4959 scope.go:122] "RemoveContainer" containerID="dcd4726bd8163208b7011947d3c280c1be50d90185c80f9e9bb21e41f911ad06" Oct 30 20:43:38.363267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:38.363247 4959 scope.go:122] "RemoveContainer" containerID="99fe084c2476bbdd4b2c3ac8eb17edf06c58b0a1ca33e9865fab942237cde292" Oct 30 20:43:38.370977 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:38.370960 4959 scope.go:122] "RemoveContainer" containerID="0d7dd70e854bb5ace0f5cdd5f8ec43f232d4c19e96df9b8689ad7cd37e7e9e7e" Oct 30 20:43:38.585685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:38.585651 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:39.966894 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:39.966859 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:41.462084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:41.462048 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:42.872782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:42.872746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:44.277764 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:44.277727 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:45.789205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:45.789159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:46.065797 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:43:46.065719 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:43:47.259760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:47.259694 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:48.760279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:48.760246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:49.027220 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:49.027144 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w"] Oct 30 20:43:49.038131 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:49.038104 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bfe6c0a-dj72w"] Oct 30 20:43:49.995281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:49.995246 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f9b3d4-1b90-4d87-80d4-9e8857a92cd4" path="/var/lib/kubelet/pods/26f9b3d4-1b90-4d87-80d4-9e8857a92cd4/volumes" Oct 30 20:43:50.272519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:50.272406 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:51.869071 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:51.869039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:53.364145 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:53.364112 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:54.868326 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:54.868297 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:56.368895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:56.368865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:57.877487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:57.877453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:59.315720 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:43:59.315686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:00.774170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:00.774113 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:02.204439 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.204390 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:02.349577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.349542 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j"] Oct 30 20:44:02.353672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.353647 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.368373 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.368352 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:44:02.368578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.368563 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:44:02.379792 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.379770 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:44:02.379901 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.379849 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:44:02.381685 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.381658 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j"] Oct 30 20:44:02.423714 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.423685 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.423852 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.423726 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.423930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.423865 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2642b\" (UniqueName: \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.525114 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.525053 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2642b\" (UniqueName: \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.525251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.525126 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.525251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.525153 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.525782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.525761 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.527665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.527643 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.543677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.543654 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2642b\" (UniqueName: \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\") pod \"maas-api-revoke-sub-05491724-7p95j\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.666519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.666468 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:02.969733 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:02.969697 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j"] Oct 30 20:44:02.974735 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:44:02.974706 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d351315_1d65_4b68_878c_b59671b55247.slice/crio-e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020 WatchSource:0}: Error finding container e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020: Status 404 returned error can't find the container with id e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020 Oct 30 20:44:03.173547 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:03.173517 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" event={"ID":"0d351315-1d65-4b68-878c-b59671b55247","Type":"ContainerStarted","Data":"e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020"} Oct 30 20:44:03.843472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:03.843407 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:04.191308 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:04.191268 4959 generic.go:356] "Generic (PLEG): container finished" podID="0d351315-1d65-4b68-878c-b59671b55247" containerID="7d40dfd172bb4793f6ce8eb9835733983c692afd4920de158b71e9fe295249b8" exitCode=0 Oct 30 20:44:04.191488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:04.191357 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" event={"ID":"0d351315-1d65-4b68-878c-b59671b55247","Type":"ContainerDied","Data":"7d40dfd172bb4793f6ce8eb9835733983c692afd4920de158b71e9fe295249b8"} Oct 30 20:44:05.331793 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.331772 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:05.454574 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.454479 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\") pod \"0d351315-1d65-4b68-878c-b59671b55247\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " Oct 30 20:44:05.454739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.454578 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\") pod \"0d351315-1d65-4b68-878c-b59671b55247\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " Oct 30 20:44:05.454739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.454612 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\" (UniqueName: \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\") pod \"0d351315-1d65-4b68-878c-b59671b55247\" (UID: \"0d351315-1d65-4b68-878c-b59671b55247\") " Oct 30 20:44:05.454990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.454959 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca" pod "0d351315-1d65-4b68-878c-b59671b55247" (UID: "0d351315-1d65-4b68-878c-b59671b55247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:05.457014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.456989 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca" pod "0d351315-1d65-4b68-878c-b59671b55247" (UID: "0d351315-1d65-4b68-878c-b59671b55247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:05.457153 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.457048 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b" pod "0d351315-1d65-4b68-878c-b59671b55247" (UID: "0d351315-1d65-4b68-878c-b59671b55247"). InnerVolumeSpecName "kube-api-access-2642b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:05.521850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.521824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:05.556078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.556052 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d351315-1d65-4b68-878c-b59671b55247-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:05.556078 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.556077 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2642b\" (UniqueName: \"kubernetes.io/projected/0d351315-1d65-4b68-878c-b59671b55247-kube-api-access-2642b\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:05.556303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:05.556087 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d351315-1d65-4b68-878c-b59671b55247-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:06.203673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:06.203636 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" event={"ID":"0d351315-1d65-4b68-878c-b59671b55247","Type":"ContainerDied","Data":"e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020"} Oct 30 20:44:06.203861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:06.203702 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c934c6de273bc26e3bcfd697d6140a7b52c10ffb8c235a4759df992569d020" Oct 30 20:44:06.203861 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:06.203654 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j" Oct 30 20:44:07.078081 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:07.078049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:08.699648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:08.699598 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:10.413301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:10.413268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:11.984893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:11.984860 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:13.166616 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.166581 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:13.432084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.432006 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh"] Oct 30 20:44:13.432629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.432605 4959 cpu_manager.go:414] "containerMap: removing container" podUID="0d351315-1d65-4b68-878c-b59671b55247" containerName="revoke-keys" Oct 30 20:44:13.432836 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.432811 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d351315-1d65-4b68-878c-b59671b55247" containerName="revoke-keys" Oct 30 20:44:13.432993 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.432977 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d351315-1d65-4b68-878c-b59671b55247" containerName="revoke-keys" Oct 30 20:44:13.436120 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.436104 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.450215 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.450194 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:44:13.461536 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.461504 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:44:13.461648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.461536 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:44:13.461648 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.461593 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:44:13.463668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.463647 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh"] Oct 30 20:44:13.529636 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.529605 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.529805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.529647 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ntj\" (UniqueName: \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.529850 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.529799 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.597863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.597835 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:13.630609 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.630580 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.630816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.630634 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.630816 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.630657 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ntj\" (UniqueName: \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.631382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.631358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.633263 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.633234 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.658602 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.658568 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ntj\" (UniqueName: \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:13.749265 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:13.749176 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:14.051343 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:14.051246 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh"] Oct 30 20:44:14.244269 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:14.244234 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" event={"ID":"1e73b970-fcb6-46c3-abe0-8793ec8b1db5","Type":"ContainerStarted","Data":"88d5c8d1a0e3d91824e74787b57e3cec358e0c35f75a2789582ae9f1cbb35e04"} Oct 30 20:44:15.251580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:15.251486 4959 generic.go:356] "Generic (PLEG): container finished" podID="1e73b970-fcb6-46c3-abe0-8793ec8b1db5" containerID="aef18a6adeb6764801c392868175ce7923776a782689b93f59b5154793f60a9b" exitCode=0 Oct 30 20:44:15.251580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:15.251559 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" event={"ID":"1e73b970-fcb6-46c3-abe0-8793ec8b1db5","Type":"ContainerDied","Data":"aef18a6adeb6764801c392868175ce7923776a782689b93f59b5154793f60a9b"} Oct 30 20:44:15.307007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:15.306979 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.066842 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:44:16.066794 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:44:16.398583 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.398561 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:16.572589 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.572561 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\" (UniqueName: \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\") pod \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " Oct 30 20:44:16.572804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.572644 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\") pod \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " Oct 30 20:44:16.572804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.572701 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\") pod \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\" (UID: \"1e73b970-fcb6-46c3-abe0-8793ec8b1db5\") " Oct 30 20:44:16.573102 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.573077 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca" pod "1e73b970-fcb6-46c3-abe0-8793ec8b1db5" (UID: "1e73b970-fcb6-46c3-abe0-8793ec8b1db5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:16.574839 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.574810 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca" pod "1e73b970-fcb6-46c3-abe0-8793ec8b1db5" (UID: "1e73b970-fcb6-46c3-abe0-8793ec8b1db5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:16.575084 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.574916 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj" pod "1e73b970-fcb6-46c3-abe0-8793ec8b1db5" (UID: "1e73b970-fcb6-46c3-abe0-8793ec8b1db5"). InnerVolumeSpecName "kube-api-access-b9ntj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:16.674129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.674093 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:16.674129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.674123 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ntj\" (UniqueName: \"kubernetes.io/projected/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-kube-api-access-b9ntj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:16.674129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.674136 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e73b970-fcb6-46c3-abe0-8793ec8b1db5-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:16.797157 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:16.797120 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:17.261882 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:17.261848 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" event={"ID":"1e73b970-fcb6-46c3-abe0-8793ec8b1db5","Type":"ContainerDied","Data":"88d5c8d1a0e3d91824e74787b57e3cec358e0c35f75a2789582ae9f1cbb35e04"} Oct 30 20:44:17.262069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:17.261897 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d5c8d1a0e3d91824e74787b57e3cec358e0c35f75a2789582ae9f1cbb35e04" Oct 30 20:44:17.262069 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:17.261856 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh" Oct 30 20:44:18.193147 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:18.193114 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:18.372258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:18.372231 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:19.872097 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:19.872058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:21.481159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:21.481121 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:22.028355 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:22.028324 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm"] Oct 30 20:44:22.039422 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:22.039393 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-m4sdm"] Oct 30 20:44:22.966983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:22.966950 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:23.606600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:23.606559 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:44:23.733592 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:23.733559 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:23.844908 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:23.844874 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:23.995042 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:23.995009 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0814b3e6-c01e-4b95-b81a-f617634a879d" path="/var/lib/kubelet/pods/0814b3e6-c01e-4b95-b81a-f617634a879d/volumes" Oct 30 20:44:24.395859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:24.395831 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:24.524721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:24.524669 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:26.612329 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:26.612295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:26.721644 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:26.721614 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:26.765138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:26.765101 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh"] Oct 30 20:44:26.775832 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:26.775804 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-051802-7rdxh"] Oct 30 20:44:27.289940 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:27.289906 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:27.995313 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:27.995269 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e73b970-fcb6-46c3-abe0-8793ec8b1db5" path="/var/lib/kubelet/pods/1e73b970-fcb6-46c3-abe0-8793ec8b1db5/volumes" Oct 30 20:44:28.607542 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:28.607482 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" containerID="cri-o://1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38" gracePeriod=25 Oct 30 20:44:32.752749 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:32.752709 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:34.009385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:34.009354 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:34.121267 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:34.121220 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:34.434371 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:34.434334 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:44:34.434629 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:34.434581 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" containerID="cri-o://e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea" gracePeriod=30 Oct 30 20:44:34.436161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:34.436101 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:35.569848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:35.569814 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:36.140293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:36.140262 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:36.836568 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:36.836540 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:44:36.838990 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:36.838953 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:44:38.918243 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:38.918197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:39.693317 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.693284 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:44:39.736484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736414 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736484 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736465 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736515 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736538 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736603 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736630 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736667 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.736718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736696 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.737062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736712 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:39.737062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736755 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\") pod \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\" (UID: \"e68b4eb8-3838-4989-8f73-0bce48eb21f1\") " Oct 30 20:44:39.737062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736839 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:39.737062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736938 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:39.737062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.736955 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:39.737305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.737179 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.737305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.737200 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.737305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.737215 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.737305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.737230 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.737305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.737288 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:39.738924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.738890 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:44:39.739070 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.739043 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "kube-api-access-xsn9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:39.739377 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.739347 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:39.739485 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.739377 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token" pod "e68b4eb8-3838-4989-8f73-0bce48eb21f1" (UID: "e68b4eb8-3838-4989-8f73-0bce48eb21f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:39.838437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.838398 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.838437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.838429 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.838437 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.838442 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsn9m\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-kube-api-access-xsn9m\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.838724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.838456 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:39.838724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:39.838469 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68b4eb8-3838-4989-8f73-0bce48eb21f1-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:40.386463 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.386432 4959 generic.go:356] "Generic (PLEG): container finished" podID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerID="e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea" exitCode=0 Oct 30 20:44:40.386971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.386486 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" Oct 30 20:44:40.386971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.386520 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerDied","Data":"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea"} Oct 30 20:44:40.386971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.386575 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2" event={"ID":"e68b4eb8-3838-4989-8f73-0bce48eb21f1","Type":"ContainerDied","Data":"eee5081127424c173c17fcc0a63ddbfc3d32380204bd498a80065cbed8f1dd43"} Oct 30 20:44:40.386971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.386596 4959 scope.go:122] "RemoveContainer" containerID="e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea" Oct 30 20:44:40.395675 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.395650 4959 scope.go:122] "RemoveContainer" containerID="5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8" Oct 30 20:44:40.403863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.403844 4959 scope.go:122] "RemoveContainer" containerID="e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea" Oct 30 20:44:40.404149 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:44:40.404130 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea\": container with ID starting with e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea not found: ID does not exist" containerID="e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea" Oct 30 20:44:40.404224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.404160 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea"} err="failed to get container status \"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea\": rpc error: code = NotFound desc = could not find container \"e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea\": container with ID starting with e57a470cb68aaa59c5a4b030ba0535421dbfd54e0c262b6c14eb8bb2a9d86dea not found: ID does not exist" Oct 30 20:44:40.404224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.404183 4959 scope.go:122] "RemoveContainer" containerID="5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8" Oct 30 20:44:40.404426 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:44:40.404408 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8\": container with ID starting with 5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8 not found: ID does not exist" containerID="5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8" Oct 30 20:44:40.404472 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.404433 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8"} err="failed to get container status \"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8\": rpc error: code = NotFound desc = could not find container \"5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8\": container with ID starting with 5c13b3f0367bf455e59b5c3545a9dc7069036d48118c4a5e9855a380e71f65a8 not found: ID does not exist" Oct 30 20:44:40.454035 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.454002 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:44:40.465292 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:40.465257 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-051802-openshift-default-86d4b98dcc8bgc2"] Oct 30 20:44:41.995018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:41.994984 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" path="/var/lib/kubelet/pods/e68b4eb8-3838-4989-8f73-0bce48eb21f1/volumes" Oct 30 20:44:42.753289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:42.753243 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:44:46.064639 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:44:46.064602 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:44:48.860017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:48.859989 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:44:48.970443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:48.970413 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\" (UniqueName: \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\") pod \"ae2d8597-deab-4050-9f26-b8b7e69aea67\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " Oct 30 20:44:48.970631 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:48.970472 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\") pod \"ae2d8597-deab-4050-9f26-b8b7e69aea67\" (UID: \"ae2d8597-deab-4050-9f26-b8b7e69aea67\") " Oct 30 20:44:48.970949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:48.970919 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume" pod "ae2d8597-deab-4050-9f26-b8b7e69aea67" (UID: "ae2d8597-deab-4050-9f26-b8b7e69aea67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:48.972760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:48.972733 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc" pod "ae2d8597-deab-4050-9f26-b8b7e69aea67" (UID: "ae2d8597-deab-4050-9f26-b8b7e69aea67"). InnerVolumeSpecName "kube-api-access-bb2xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:49.071939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.071888 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2xc\" (UniqueName: \"kubernetes.io/projected/ae2d8597-deab-4050-9f26-b8b7e69aea67-kube-api-access-bb2xc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:49.071939 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.071920 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae2d8597-deab-4050-9f26-b8b7e69aea67-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:44:49.475983 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.475951 4959 generic.go:356] "Generic (PLEG): container finished" podID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerID="1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38" exitCode=0 Oct 30 20:44:49.476194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.475994 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" Oct 30 20:44:49.476194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.476029 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" event={"ID":"ae2d8597-deab-4050-9f26-b8b7e69aea67","Type":"ContainerDied","Data":"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38"} Oct 30 20:44:49.476194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.476082 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc" event={"ID":"ae2d8597-deab-4050-9f26-b8b7e69aea67","Type":"ContainerDied","Data":"6bfb0bdb4510007e10cb26cd0acfa78d064bdbd9354e27717b1453e3a1b3aac7"} Oct 30 20:44:49.476194 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.476104 4959 scope.go:122] "RemoveContainer" containerID="1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38" Oct 30 20:44:49.486713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.486698 4959 scope.go:122] "RemoveContainer" containerID="1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38" Oct 30 20:44:49.486947 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:44:49.486910 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38\": container with ID starting with 1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38 not found: ID does not exist" containerID="1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38" Oct 30 20:44:49.487052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.486944 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38"} err="failed to get container status \"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38\": rpc error: code = NotFound desc = could not find container \"1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38\": container with ID starting with 1a262bba4e45be4c4fb46ec8708fa1f796cc67060d307dc356f0bfe7621f8a38 not found: ID does not exist" Oct 30 20:44:49.526557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.526524 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:44:49.538156 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.538131 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-051802-7c8bb52vfc"] Oct 30 20:44:49.994801 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:49.994765 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" path="/var/lib/kubelet/pods/ae2d8597-deab-4050-9f26-b8b7e69aea67/volumes" Oct 30 20:44:56.025663 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:56.025629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.401948 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:56.401918 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.748161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:44:56.748083 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:00.214476 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.214434 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:00.215086 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215058 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1e73b970-fcb6-46c3-abe0-8793ec8b1db5" containerName="revoke-keys" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215088 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e73b970-fcb6-46c3-abe0-8793ec8b1db5" containerName="revoke-keys" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215110 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215119 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215135 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215144 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215166 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215174 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215328 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215342 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2d8597-deab-4050-9f26-b8b7e69aea67" containerName="payload-pre-processing" Oct 30 20:45:00.215681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215354 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68b4eb8-3838-4989-8f73-0bce48eb21f1" containerName="istio-proxy" Oct 30 20:45:00.215681 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.215372 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e73b970-fcb6-46c3-abe0-8793ec8b1db5" containerName="revoke-keys" Oct 30 20:45:00.220587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.220563 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.237954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.237916 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:00.279673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.279631 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:45:00.279921 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.279743 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:45:00.284453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.284427 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:45:00.391964 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.391919 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.392299 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.392274 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskln\" (UniqueName: \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.493149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.493040 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.493452 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.493423 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskln\" (UniqueName: \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.494558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.494399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.537668 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.537625 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskln\" (UniqueName: \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\") pod \"maas-api-key-cleanup-29889885-vdjz6\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:00.540789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:00.540761 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:01.174673 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:01.174632 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:01.299418 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:45:01.299370 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf440fb9f_d362_4576_b69e_5a65e26d199a.slice/crio-0cdc17739a4b88eba2123eebaefc5b39e01f4c9e158d58bbdec70ba5184daca8 WatchSource:0}: Error finding container 0cdc17739a4b88eba2123eebaefc5b39e01f4c9e158d58bbdec70ba5184daca8: Status 404 returned error can't find the container with id 0cdc17739a4b88eba2123eebaefc5b39e01f4c9e158d58bbdec70ba5184daca8 Oct 30 20:45:01.572594 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:01.562567 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerStarted","Data":"0cdc17739a4b88eba2123eebaefc5b39e01f4c9e158d58bbdec70ba5184daca8"} Oct 30 20:45:02.614521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:02.600741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.614521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:02.610839 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerStarted","Data":"76d60e4536b2ab987f6f2e20966e8473374528b42001415f7712a4e1c57271cb"} Oct 30 20:45:02.720517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:02.720467 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.006208 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:03.006172 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.617672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:03.617636 4959 generic.go:356] "Generic (PLEG): container finished" podID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerID="76d60e4536b2ab987f6f2e20966e8473374528b42001415f7712a4e1c57271cb" exitCode=60 Oct 30 20:45:03.618121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:03.617718 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerDied","Data":"76d60e4536b2ab987f6f2e20966e8473374528b42001415f7712a4e1c57271cb"} Oct 30 20:45:03.618121 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:03.617914 4959 scope.go:122] "RemoveContainer" containerID="76d60e4536b2ab987f6f2e20966e8473374528b42001415f7712a4e1c57271cb" Oct 30 20:45:04.623955 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:04.623914 4959 generic.go:356] "Generic (PLEG): container finished" podID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerID="a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6" exitCode=60 Oct 30 20:45:04.624363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:04.624005 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerDied","Data":"a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6"} Oct 30 20:45:04.624363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:04.624073 4959 scope.go:122] "RemoveContainer" containerID="76d60e4536b2ab987f6f2e20966e8473374528b42001415f7712a4e1c57271cb" Oct 30 20:45:04.624363 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:04.624209 4959 scope.go:122] "RemoveContainer" containerID="a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6" Oct 30 20:45:04.624541 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:45:04.624472 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-vdjz6_odh-ai-gateway-infra(f440fb9f-d362-4576-b69e-5a65e26d199a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" Oct 30 20:45:05.630564 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:05.630535 4959 scope.go:122] "RemoveContainer" containerID="a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6" Oct 30 20:45:05.630996 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:45:05.630762 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-vdjz6_odh-ai-gateway-infra(f440fb9f-d362-4576-b69e-5a65e26d199a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" Oct 30 20:45:16.213525 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:45:16.210701 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:45:17.989053 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:17.989026 4959 scope.go:122] "RemoveContainer" containerID="a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6" Oct 30 20:45:18.747717 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:18.747679 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerStarted","Data":"7c3c9236cac989ff10bed83ed6b355d0cfc3b1d825f401e7eb1f03386dc00bea"} Oct 30 20:45:18.793423 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:18.793314 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" podStartSLOduration=18.79328812 podStartE2EDuration="18.79328812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:02.661724798 +0000 UTC m=+4226.790958720" watchObservedRunningTime="2026-10-30 20:45:18.79328812 +0000 UTC m=+4242.922522042" Oct 30 20:45:19.754285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:19.754250 4959 generic.go:356] "Generic (PLEG): container finished" podID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerID="7c3c9236cac989ff10bed83ed6b355d0cfc3b1d825f401e7eb1f03386dc00bea" exitCode=60 Oct 30 20:45:19.754804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:19.754328 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerDied","Data":"7c3c9236cac989ff10bed83ed6b355d0cfc3b1d825f401e7eb1f03386dc00bea"} Oct 30 20:45:19.754804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:19.754390 4959 scope.go:122] "RemoveContainer" containerID="a790ed4cc9e632802124813734cac16bca3278574c0c87d4faf890830ad6b3f6" Oct 30 20:45:19.754804 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:19.754734 4959 scope.go:122] "RemoveContainer" containerID="7c3c9236cac989ff10bed83ed6b355d0cfc3b1d825f401e7eb1f03386dc00bea" Oct 30 20:45:19.755043 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:45:19.755023 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-vdjz6_odh-ai-gateway-infra(f440fb9f-d362-4576-b69e-5a65e26d199a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" Oct 30 20:45:19.799627 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:19.799600 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:20.901201 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:20.901171 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:20.947751 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:20.947716 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\") pod \"f440fb9f-d362-4576-b69e-5a65e26d199a\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " Oct 30 20:45:20.948005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:20.947789 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\" (UniqueName: \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\") pod \"f440fb9f-d362-4576-b69e-5a65e26d199a\" (UID: \"f440fb9f-d362-4576-b69e-5a65e26d199a\") " Oct 30 20:45:20.948584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:20.948548 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca" pod "f440fb9f-d362-4576-b69e-5a65e26d199a" (UID: "f440fb9f-d362-4576-b69e-5a65e26d199a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:20.950828 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:20.950796 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln" pod "f440fb9f-d362-4576-b69e-5a65e26d199a" (UID: "f440fb9f-d362-4576-b69e-5a65e26d199a"). InnerVolumeSpecName "kube-api-access-qskln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:21.048972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.048883 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f440fb9f-d362-4576-b69e-5a65e26d199a-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:21.048972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.048908 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskln\" (UniqueName: \"kubernetes.io/projected/f440fb9f-d362-4576-b69e-5a65e26d199a-kube-api-access-qskln\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:21.766747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.766715 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" Oct 30 20:45:21.766965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.766753 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6" event={"ID":"f440fb9f-d362-4576-b69e-5a65e26d199a","Type":"ContainerDied","Data":"0cdc17739a4b88eba2123eebaefc5b39e01f4c9e158d58bbdec70ba5184daca8"} Oct 30 20:45:21.766965 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.766815 4959 scope.go:122] "RemoveContainer" containerID="7c3c9236cac989ff10bed83ed6b355d0cfc3b1d825f401e7eb1f03386dc00bea" Oct 30 20:45:21.817849 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.817824 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:21.829124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.829099 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-vdjz6"] Oct 30 20:45:21.994350 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:21.994315 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" path="/var/lib/kubelet/pods/f440fb9f-d362-4576-b69e-5a65e26d199a/volumes" Oct 30 20:45:23.834665 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:23.834635 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:29.120522 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:29.118278 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:33.109979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.109951 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2"] Oct 30 20:45:33.110460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110431 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110460 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110450 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110473 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110480 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110627 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110642 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.110698 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.110655 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:33.115044 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.115017 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.140095 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.140069 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.140228 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.140206 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzlj\" (UniqueName: \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.140279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.140249 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.141707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.141685 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:45:33.141833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.141710 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:45:33.141833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.141758 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:45:33.141833 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.141801 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:45:33.143395 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.143372 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2"] Oct 30 20:45:33.241600 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.241577 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.241703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.241640 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzlj\" (UniqueName: \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.241703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.241658 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.242238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.242216 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.244216 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.244195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.260782 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.260759 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzlj\" (UniqueName: \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\") pod \"maas-api-revoke-sub-3e31a506-mszb2\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.427731 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.427714 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:33.725205 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.725120 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2"] Oct 30 20:45:33.730285 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:45:33.730256 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b5a730_c9ed_49dc_885c_86e1a5ee3b29.slice/crio-9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e WatchSource:0}: Error finding container 9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e: Status 404 returned error can't find the container with id 9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e Oct 30 20:45:33.874817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:33.874790 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" event={"ID":"67b5a730-c9ed-49dc-885c-86e1a5ee3b29","Type":"ContainerStarted","Data":"9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e"} Oct 30 20:45:34.884586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:34.884545 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" event={"ID":"67b5a730-c9ed-49dc-885c-86e1a5ee3b29","Type":"ContainerStarted","Data":"9f61aed494f9a5ba41c1fb987e5873b24afe58e968c6dc4952a01dcb2b0745b3"} Oct 30 20:45:34.921457 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:34.921401 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" podStartSLOduration=1.921385605 podStartE2EDuration="1.921385605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:34.920821164 +0000 UTC m=+4259.050055087" watchObservedRunningTime="2026-10-30 20:45:34.921385605 +0000 UTC m=+4259.050619526" Oct 30 20:45:35.809520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:35.808746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:35.916520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:35.915535 4959 generic.go:356] "Generic (PLEG): container finished" podID="67b5a730-c9ed-49dc-885c-86e1a5ee3b29" containerID="9f61aed494f9a5ba41c1fb987e5873b24afe58e968c6dc4952a01dcb2b0745b3" exitCode=0 Oct 30 20:45:35.916520 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:35.915681 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" event={"ID":"67b5a730-c9ed-49dc-885c-86e1a5ee3b29","Type":"ContainerDied","Data":"9f61aed494f9a5ba41c1fb987e5873b24afe58e968c6dc4952a01dcb2b0745b3"} Oct 30 20:45:36.424015 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:36.423982 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:37.113539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.113517 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:37.203736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.203677 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\") pod \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " Oct 30 20:45:37.203736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.203724 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\" (UniqueName: \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\") pod \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " Oct 30 20:45:37.203922 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.203772 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\") pod \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\" (UID: \"67b5a730-c9ed-49dc-885c-86e1a5ee3b29\") " Oct 30 20:45:37.204007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.203986 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca" pod "67b5a730-c9ed-49dc-885c-86e1a5ee3b29" (UID: "67b5a730-c9ed-49dc-885c-86e1a5ee3b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:37.204087 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.204062 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:37.205902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.205880 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca" pod "67b5a730-c9ed-49dc-885c-86e1a5ee3b29" (UID: "67b5a730-c9ed-49dc-885c-86e1a5ee3b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:37.206010 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.205992 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj" pod "67b5a730-c9ed-49dc-885c-86e1a5ee3b29" (UID: "67b5a730-c9ed-49dc-885c-86e1a5ee3b29"). InnerVolumeSpecName "kube-api-access-5dzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:37.305330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.305307 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzlj\" (UniqueName: \"kubernetes.io/projected/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-kube-api-access-5dzlj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:37.305330 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.305329 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b5a730-c9ed-49dc-885c-86e1a5ee3b29-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:37.930891 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.930858 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" Oct 30 20:45:37.931074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.930886 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2" event={"ID":"67b5a730-c9ed-49dc-885c-86e1a5ee3b29","Type":"ContainerDied","Data":"9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e"} Oct 30 20:45:37.931074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:37.930946 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddf4d73deadc77fba732d53909f3d9498604b4835692f65a16895fef816054e" Oct 30 20:45:38.405310 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:38.405278 4959 scope.go:122] "RemoveContainer" containerID="f4e074c08fa6d265a39fda0d461f873f5ba95805131f10273f98d92b4eb0a4dc" Oct 30 20:45:38.415456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:38.415434 4959 scope.go:122] "RemoveContainer" containerID="a3926e3794e525a324cee9085940a4d30f0ff3460efa9c5d68e49624a6aa75f2" Oct 30 20:45:38.424335 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:38.424314 4959 scope.go:122] "RemoveContainer" containerID="228130f98e76156458e5ea3662f2465fc91ac35c2e9c13959e5268044ba92d1a" Oct 30 20:45:46.064847 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:45:46.064810 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:45:53.513604 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.513570 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw"] Oct 30 20:45:53.514161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.514038 4959 cpu_manager.go:414] "containerMap: removing container" podUID="67b5a730-c9ed-49dc-885c-86e1a5ee3b29" containerName="revoke-keys" Oct 30 20:45:53.514161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.514054 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b5a730-c9ed-49dc-885c-86e1a5ee3b29" containerName="revoke-keys" Oct 30 20:45:53.514161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.514095 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:53.514161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.514100 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f440fb9f-d362-4576-b69e-5a65e26d199a" containerName="cleanup" Oct 30 20:45:53.514328 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.514171 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b5a730-c9ed-49dc-885c-86e1a5ee3b29" containerName="revoke-keys" Oct 30 20:45:53.517301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.517269 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.531217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.531192 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:45:53.531372 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.531350 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8nlg\"" Oct 30 20:45:53.542356 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.542332 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:45:53.542521 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.542371 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:45:53.545268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.545241 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw"] Oct 30 20:45:53.686311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.686265 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.686525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.686345 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsb94\" (UniqueName: \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.686525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.686446 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.787302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.787213 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsb94\" (UniqueName: \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.787302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.787264 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.787534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.787305 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.788117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.788034 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.789866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.789844 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.809859 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.809831 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsb94\" (UniqueName: \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\") pod \"maas-api-revoke-sub-d3feede2-shkgw\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:53.830288 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:53.830269 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:54.149863 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:54.149816 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw"] Oct 30 20:45:54.154575 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:45:54.154550 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed307e6_da08_4266_9c4a_abe836d1a5c1.slice/crio-5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7 WatchSource:0}: Error finding container 5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7: Status 404 returned error can't find the container with id 5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7 Oct 30 20:45:55.053488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:55.053454 4959 generic.go:356] "Generic (PLEG): container finished" podID="9ed307e6-da08-4266-9c4a-abe836d1a5c1" containerID="acf7bb5ce4d1f8afd0821bddc485ea8ac52aaaa3dd2587bd92b2c0ddbe45fe63" exitCode=0 Oct 30 20:45:55.053877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:55.053548 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" event={"ID":"9ed307e6-da08-4266-9c4a-abe836d1a5c1","Type":"ContainerDied","Data":"acf7bb5ce4d1f8afd0821bddc485ea8ac52aaaa3dd2587bd92b2c0ddbe45fe63"} Oct 30 20:45:55.053877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:55.053606 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" event={"ID":"9ed307e6-da08-4266-9c4a-abe836d1a5c1","Type":"ContainerStarted","Data":"5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7"} Oct 30 20:45:56.190671 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.190640 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:56.314256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.314167 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\") pod \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " Oct 30 20:45:56.314256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.314210 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\") pod \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " Oct 30 20:45:56.314256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.314248 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\" (UniqueName: \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\") pod \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\" (UID: \"9ed307e6-da08-4266-9c4a-abe836d1a5c1\") " Oct 30 20:45:56.314712 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.314587 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca" pod "9ed307e6-da08-4266-9c4a-abe836d1a5c1" (UID: "9ed307e6-da08-4266-9c4a-abe836d1a5c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:56.316443 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.316418 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca" pod "9ed307e6-da08-4266-9c4a-abe836d1a5c1" (UID: "9ed307e6-da08-4266-9c4a-abe836d1a5c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:56.316586 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.316505 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94" pod "9ed307e6-da08-4266-9c4a-abe836d1a5c1" (UID: "9ed307e6-da08-4266-9c4a-abe836d1a5c1"). InnerVolumeSpecName "kube-api-access-wsb94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:56.415713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.415680 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-xks-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:56.415713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.415709 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed307e6-da08-4266-9c4a-abe836d1a5c1-maas-api-openshift-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:56.415713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:56.415720 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsb94\" (UniqueName: \"kubernetes.io/projected/9ed307e6-da08-4266-9c4a-abe836d1a5c1-kube-api-access-wsb94\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:45:57.063304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:57.063270 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" event={"ID":"9ed307e6-da08-4266-9c4a-abe836d1a5c1","Type":"ContainerDied","Data":"5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7"} Oct 30 20:45:57.063304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:57.063286 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw" Oct 30 20:45:57.063544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:45:57.063317 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e37c6c51b57bc3604f9158a5068f932711efd060174f0627a4e45fee78e33f7" Oct 30 20:46:06.066174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.066136 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:06.066817 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.066796 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9ed307e6-da08-4266-9c4a-abe836d1a5c1" containerName="revoke-keys" Oct 30 20:46:06.066871 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.066821 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed307e6-da08-4266-9c4a-abe836d1a5c1" containerName="revoke-keys" Oct 30 20:46:06.066976 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.066964 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed307e6-da08-4266-9c4a-abe836d1a5c1" containerName="revoke-keys" Oct 30 20:46:06.070760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.070735 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.098348 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.098313 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:46:06.098534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.098406 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pcrrd\"" Oct 30 20:46:06.098534 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.098463 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:46:06.098667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.098617 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:46:06.099966 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.099943 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:46:06.100575 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.100551 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.100723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.100615 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bz4\" (UniqueName: \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.100723 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.100699 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.100842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.100755 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.102076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.102058 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:06.202124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.202084 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bz4\" (UniqueName: \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.202124 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.202124 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.202401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.202149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.202401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.202202 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.202945 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.202925 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.204892 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.204868 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.205038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.204906 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.220927 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.220905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bz4\" (UniqueName: \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\") pod \"maas-api-f7476c9bf-vrj6t\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.387579 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.387559 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:06.694441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.694361 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:06.699364 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:46:06.699333 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a54d3a6_793a_4900_8888_dce4b08e6048.slice/crio-5388c80c257ad706cad882a4226329752bf7953c3b54e81e8009e2691a16e1a5 WatchSource:0}: Error finding container 5388c80c257ad706cad882a4226329752bf7953c3b54e81e8009e2691a16e1a5: Status 404 returned error can't find the container with id 5388c80c257ad706cad882a4226329752bf7953c3b54e81e8009e2691a16e1a5 Oct 30 20:46:06.700642 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:06.700621 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:46:07.113661 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:07.113627 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" event={"ID":"4a54d3a6-793a-4900-8888-dce4b08e6048","Type":"ContainerStarted","Data":"5388c80c257ad706cad882a4226329752bf7953c3b54e81e8009e2691a16e1a5"} Oct 30 20:46:08.120188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:08.120155 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" event={"ID":"4a54d3a6-793a-4900-8888-dce4b08e6048","Type":"ContainerStarted","Data":"2537073c25a297b8bfd89451bdb54b793af7b073889bde25b8e102302d9c10e3"} Oct 30 20:46:08.120656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:08.120226 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:08.162657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:08.162594 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" podStartSLOduration=1.762481838 podStartE2EDuration="2.162578821s" totalImagesPullingTime="400.096983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:08.160603536 +0000 UTC m=+4292.289837456" watchObservedRunningTime="2026-10-30 20:46:08.162578821 +0000 UTC m=+4292.291812791" Oct 30 20:46:11.048677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:11.048642 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9"] Oct 30 20:46:11.059747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:11.059717 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-kd5d9"] Oct 30 20:46:11.995741 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:11.995710 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e739245-6510-4d14-9bb1-472a25715f11" path="/var/lib/kubelet/pods/9e739245-6510-4d14-9bb1-472a25715f11/volumes" Oct 30 20:46:13.016703 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.016670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:13.411561 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.411524 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:13.416278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.416251 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.434553 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.434532 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f5c2161a-openshift-default-dockercfg-d6llj\"" Oct 30 20:46:13.448361 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.448336 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:13.464179 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464154 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464195 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464221 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464359 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464440 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464413 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464548 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqn8\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464607 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464707 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464690 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.464747 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.464724 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.567614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.567573 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.568293 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.568263 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569001 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.568951 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569279 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569262 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569436 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569332 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.568872 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569420 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569551 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569586 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569634 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.569842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569673 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqn8\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.570018 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.569992 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.570311 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.570281 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.570418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.570336 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.573127 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.573099 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.582273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.582247 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.587842 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.587820 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.588047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.588027 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqn8\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\") pod \"e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.730532 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.730413 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:13.934512 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.934462 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:13.934760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.934729 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" podUID="4a54d3a6-793a-4900-8888-dce4b08e6048" containerName="maas-api" containerID="cri-o://2537073c25a297b8bfd89451bdb54b793af7b073889bde25b8e102302d9c10e3" gracePeriod=30 Oct 30 20:46:13.940488 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:13.940457 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:14.003411 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.003331 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf"] Oct 30 20:46:14.007307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.007278 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.042442 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.042419 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf"] Oct 30 20:46:14.064760 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.064674 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:14.067573 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:46:14.067547 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fe8141_aeb6_4785_8274_1c836da79637.slice/crio-475db9fafd43f320296ad5789c572fbbc9b0e625fcc983b98bc2d381c5690151 WatchSource:0}: Error finding container 475db9fafd43f320296ad5789c572fbbc9b0e625fcc983b98bc2d381c5690151: Status 404 returned error can't find the container with id 475db9fafd43f320296ad5789c572fbbc9b0e625fcc983b98bc2d381c5690151 Oct 30 20:46:14.070007 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.069972 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:14.070126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.070052 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:14.070126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.070088 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:14.074895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.074871 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-metrics-certs\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.075113 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.075081 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4nj\" (UniqueName: \"kubernetes.io/projected/b4a0d58f-ce59-474f-85f2-e69fff014d24-kube-api-access-gg4nj\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.075251 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.075227 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-maas-api-tls\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.075381 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.075353 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0d58f-ce59-474f-85f2-e69fff014d24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.108847 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.108822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:14.150915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.150787 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerStarted","Data":"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b"} Oct 30 20:46:14.150915 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.150853 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerStarted","Data":"475db9fafd43f320296ad5789c572fbbc9b0e625fcc983b98bc2d381c5690151"} Oct 30 20:46:14.160530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.152904 4959 generic.go:356] "Generic (PLEG): container finished" podID="4a54d3a6-793a-4900-8888-dce4b08e6048" containerID="2537073c25a297b8bfd89451bdb54b793af7b073889bde25b8e102302d9c10e3" exitCode=0 Oct 30 20:46:14.160530 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.157295 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" event={"ID":"4a54d3a6-793a-4900-8888-dce4b08e6048","Type":"ContainerDied","Data":"2537073c25a297b8bfd89451bdb54b793af7b073889bde25b8e102302d9c10e3"} Oct 30 20:46:14.177278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.176450 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-maas-api-tls\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.177278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.176513 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0d58f-ce59-474f-85f2-e69fff014d24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.177278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.176612 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-metrics-certs\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.177278 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.176705 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4nj\" (UniqueName: \"kubernetes.io/projected/b4a0d58f-ce59-474f-85f2-e69fff014d24-kube-api-access-gg4nj\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.178345 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.178314 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0d58f-ce59-474f-85f2-e69fff014d24-openshift-service-ca\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.180027 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.180000 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-metrics-certs\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.180582 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.180552 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4a0d58f-ce59-474f-85f2-e69fff014d24-maas-api-tls\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.190571 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.190549 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:14.195585 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.195524 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podStartSLOduration=1.195507662 podStartE2EDuration="1.195507662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:14.193820383 +0000 UTC m=+4298.323054315" watchObservedRunningTime="2026-10-30 20:46:14.195507662 +0000 UTC m=+4298.324741579" Oct 30 20:46:14.197865 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.197846 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4nj\" (UniqueName: \"kubernetes.io/projected/b4a0d58f-ce59-474f-85f2-e69fff014d24-kube-api-access-gg4nj\") pod \"maas-api-76f6f44c6f-8k9rf\" (UID: \"b4a0d58f-ce59-474f-85f2-e69fff014d24\") " pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.277576 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.277473 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\") pod \"4a54d3a6-793a-4900-8888-dce4b08e6048\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " Oct 30 20:46:14.277744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.277639 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\") pod \"4a54d3a6-793a-4900-8888-dce4b08e6048\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " Oct 30 20:46:14.277744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.277689 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\") pod \"4a54d3a6-793a-4900-8888-dce4b08e6048\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " Oct 30 20:46:14.277744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.277725 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\" (UniqueName: \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\") pod \"4a54d3a6-793a-4900-8888-dce4b08e6048\" (UID: \"4a54d3a6-793a-4900-8888-dce4b08e6048\") " Oct 30 20:46:14.277951 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.277881 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca" pod "4a54d3a6-793a-4900-8888-dce4b08e6048" (UID: "4a54d3a6-793a-4900-8888-dce4b08e6048"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:14.278129 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.278103 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a54d3a6-793a-4900-8888-dce4b08e6048-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:14.280209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.280187 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs" pod "4a54d3a6-793a-4900-8888-dce4b08e6048" (UID: "4a54d3a6-793a-4900-8888-dce4b08e6048"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:14.280290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.280211 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4" pod "4a54d3a6-793a-4900-8888-dce4b08e6048" (UID: "4a54d3a6-793a-4900-8888-dce4b08e6048"). InnerVolumeSpecName "kube-api-access-94bz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:14.280387 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.280367 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls" pod "4a54d3a6-793a-4900-8888-dce4b08e6048" (UID: "4a54d3a6-793a-4900-8888-dce4b08e6048"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:14.329429 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.329399 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:14.379401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.379135 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:14.379401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.379177 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bz4\" (UniqueName: \"kubernetes.io/projected/4a54d3a6-793a-4900-8888-dce4b08e6048-kube-api-access-94bz4\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:14.379401 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.379193 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a54d3a6-793a-4900-8888-dce4b08e6048-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:14.668906 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.668850 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf"] Oct 30 20:46:14.673686 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:46:14.673657 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a0d58f_ce59_474f_85f2_e69fff014d24.slice/crio-a708705d7cfa1eb7f4486ec9d24d61c849a284f8797f15879904e9a3fc46ba3c WatchSource:0}: Error finding container a708705d7cfa1eb7f4486ec9d24d61c849a284f8797f15879904e9a3fc46ba3c: Status 404 returned error can't find the container with id a708705d7cfa1eb7f4486ec9d24d61c849a284f8797f15879904e9a3fc46ba3c Oct 30 20:46:14.730634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:14.730609 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:15.166170 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.166134 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" event={"ID":"b4a0d58f-ce59-474f-85f2-e69fff014d24","Type":"ContainerStarted","Data":"15cbf025eabffe3857e82b56cabea13594218e82e1f157f0fb0d674050dd8dda"} Oct 30 20:46:15.166730 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.166198 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" event={"ID":"b4a0d58f-ce59-474f-85f2-e69fff014d24","Type":"ContainerStarted","Data":"a708705d7cfa1eb7f4486ec9d24d61c849a284f8797f15879904e9a3fc46ba3c"} Oct 30 20:46:15.166730 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.166315 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:15.168199 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.168180 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" Oct 30 20:46:15.168324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.168216 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t" event={"ID":"4a54d3a6-793a-4900-8888-dce4b08e6048","Type":"ContainerDied","Data":"5388c80c257ad706cad882a4226329752bf7953c3b54e81e8009e2691a16e1a5"} Oct 30 20:46:15.168324 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.168262 4959 scope.go:122] "RemoveContainer" containerID="2537073c25a297b8bfd89451bdb54b793af7b073889bde25b8e102302d9c10e3" Oct 30 20:46:15.206303 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.206211 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" podStartSLOduration=1.933678069 podStartE2EDuration="2.206195751s" totalImagesPullingTime="272.517682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:15.204348183 +0000 UTC m=+4299.333582104" watchObservedRunningTime="2026-10-30 20:46:15.206195751 +0000 UTC m=+4299.335429674" Oct 30 20:46:15.238382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.238351 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:15.241808 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.241789 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7476c9bf-vrj6t"] Oct 30 20:46:15.736664 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.736638 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:15.816959 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.816927 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.939588 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.939553 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:15.995686 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:15.995592 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a54d3a6-793a-4900-8888-dce4b08e6048" path="/var/lib/kubelet/pods/4a54d3a6-793a-4900-8888-dce4b08e6048/volumes" Oct 30 20:46:16.065296 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:46:16.065260 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:46:16.174656 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:16.174622 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:16.175743 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:16.175726 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:17.671755 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:17.671717 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:17.784888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:17.784855 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:18.923464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:18.923428 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:19.049233 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:19.049198 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:20.626307 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:20.626270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:21.180306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:21.180277 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76f6f44c6f-8k9rf" Oct 30 20:46:22.183905 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:22.183869 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:23.785148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:23.785116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:25.282973 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:25.282928 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:26.680590 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:26.680550 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:28.086174 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:28.086136 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:29.585144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:29.585102 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:31.091180 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:31.091145 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:31.977275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:31.976481 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": read tcp 10.132.0.2:56276->10.132.0.255:15021: read: connection reset by peer" start-of-body= Oct 30 20:46:31.977275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:31.976575 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": read tcp 10.132.0.2:56276->10.132.0.255:15021: read: connection reset by peer" Oct 30 20:46:32.257020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.256931 4959 generic.go:356] "Generic (PLEG): container finished" podID="d3fe8141-aeb6-4785-8274-1c836da79637" containerID="c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b" exitCode=0 Oct 30 20:46:32.257020 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.257006 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerDied","Data":"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b"} Oct 30 20:46:32.257573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.257375 4959 scope.go:122] "RemoveContainer" containerID="c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b" Oct 30 20:46:32.258622 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.258589 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:32.258718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.258665 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:32.258718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.258700 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:46:32.746573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:32.746146 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:33.278238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:33.278200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerStarted","Data":"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074"} Oct 30 20:46:33.730712 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:33.730674 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:33.731088 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:33.731062 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:46:33.731204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:33.731120 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:46:34.292790 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:34.292754 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:34.731862 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:34.731824 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:46:34.732106 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:34.731902 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:46:35.416270 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:35.416234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:35.731487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:35.731409 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:46:35.731487 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:35.731473 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:46:35.779380 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:35.779346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:35.906625 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:35.906586 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:36.470592 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:36.470533 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:36.731159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:36.731073 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:46:36.731159 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:36.731124 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:46:37.260032 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:37.259951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:37.731285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:37.731251 4959 patch_prober.go:29] interesting pod/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:46:37.731735 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:37.731334 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:46:38.686340 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:38.686310 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:38.748480 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:38.748396 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:38.749918 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:38.749885 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:38.755580 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:38.752550 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:40.183537 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:40.183460 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:41.685578 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:41.685543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:41.986818 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:41.986732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:42.094539 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:42.093983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:42.901506 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:42.901473 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.365688 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:43.365609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:43.473482 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:43.473451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:46.064536 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:46:46.064479 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:46:48.055160 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:48.055126 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:48.055709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:48.055358 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" containerID="cri-o://10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074" gracePeriod=30 Oct 30 20:46:48.102000 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:48.101969 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:49.426402 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:49.426369 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:49.989525 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:49.989478 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:50.105477 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:50.105441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:53.315784 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.315759 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:53.383057 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.383021 4959 generic.go:356] "Generic (PLEG): container finished" podID="d3fe8141-aeb6-4785-8274-1c836da79637" containerID="10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074" exitCode=0 Oct 30 20:46:53.383248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.383158 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" Oct 30 20:46:53.383248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.383171 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerDied","Data":"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074"} Oct 30 20:46:53.383248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.383221 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj" event={"ID":"d3fe8141-aeb6-4785-8274-1c836da79637","Type":"ContainerDied","Data":"475db9fafd43f320296ad5789c572fbbc9b0e625fcc983b98bc2d381c5690151"} Oct 30 20:46:53.383444 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.383250 4959 scope.go:122] "RemoveContainer" containerID="10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074" Oct 30 20:46:53.392775 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.392751 4959 scope.go:122] "RemoveContainer" containerID="c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b" Oct 30 20:46:53.401273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.401249 4959 scope.go:122] "RemoveContainer" containerID="10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074" Oct 30 20:46:53.401585 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:46:53.401564 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074\": container with ID starting with 10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074 not found: ID does not exist" containerID="10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074" Oct 30 20:46:53.401657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.401594 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074"} err="failed to get container status \"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074\": rpc error: code = NotFound desc = could not find container \"10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074\": container with ID starting with 10aaacc849c56ea0fd2b041d10d6667f9c8a29995b497333ebdec34474170074 not found: ID does not exist" Oct 30 20:46:53.401657 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.401613 4959 scope.go:122] "RemoveContainer" containerID="c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b" Oct 30 20:46:53.401904 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:46:53.401887 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b\": container with ID starting with c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b not found: ID does not exist" containerID="c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b" Oct 30 20:46:53.401957 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.401910 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b"} err="failed to get container status \"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b\": rpc error: code = NotFound desc = could not find container \"c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b\": container with ID starting with c48d46b18477517bb9fecc65c3f0a20ef7be5b2bbd34b2eea6649c64cba5110b not found: ID does not exist" Oct 30 20:46:53.416272 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416241 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416288 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416324 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416434 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416387 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416443 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416486 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416634 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416578 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:53.416810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416654 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416689 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416743 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\") pod \"d3fe8141-aeb6-4785-8274-1c836da79637\" (UID: \"d3fe8141-aeb6-4785-8274-1c836da79637\") " Oct 30 20:46:53.416984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416818 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:46:53.416984 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416957 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:53.417096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.416974 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:53.417096 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417041 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:53.417306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417210 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.417306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417235 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.417306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417251 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.417306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417265 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.417306 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.417278 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3fe8141-aeb6-4785-8274-1c836da79637-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.419038 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.419008 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "kube-api-access-7kqn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:53.419139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.419053 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:46:53.419139 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.419107 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:46:53.419273 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.419258 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token" pod "d3fe8141-aeb6-4785-8274-1c836da79637" (UID: "d3fe8141-aeb6-4785-8274-1c836da79637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:53.518023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.517984 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.518023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.518017 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqn8\" (UniqueName: \"kubernetes.io/projected/d3fe8141-aeb6-4785-8274-1c836da79637-kube-api-access-7kqn8\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.518023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.518027 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3fe8141-aeb6-4785-8274-1c836da79637-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.518023 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.518037 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3fe8141-aeb6-4785-8274-1c836da79637-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:46:53.628144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.628101 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:53.733898 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.733859 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:53.743508 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.743469 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f5c2161a-openshift-default-5b4f4b4679-fs7cj"] Oct 30 20:46:53.766667 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.766634 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:53.995211 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:53.995132 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" path="/var/lib/kubelet/pods/d3fe8141-aeb6-4785-8274-1c836da79637/volumes" Oct 30 20:46:54.956418 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956363 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956845 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4a54d3a6-793a-4900-8888-dce4b08e6048" containerName="maas-api" Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956860 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a54d3a6-793a-4900-8888-dce4b08e6048" containerName="maas-api" Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956890 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956895 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956902 4959 cpu_manager.go:414] "containerMap: removing container" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.956907 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956907 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.957168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956973 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.957168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956982 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fe8141-aeb6-4785-8274-1c836da79637" containerName="istio-proxy" Oct 30 20:46:54.957168 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.956990 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a54d3a6-793a-4900-8888-dce4b08e6048" containerName="maas-api" Oct 30 20:46:54.961810 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.961787 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:54.978210 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.978175 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-72e6e5-serving-cert\"" Oct 30 20:46:54.991241 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:54.991207 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:46:55.036188 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.036156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:55.135684 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.135639 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mtj\" (UniqueName: \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.135902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.135799 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.135978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.135916 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.135978 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.135962 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.155893 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.155852 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:55.236729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.236628 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.236729 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.236668 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mtj\" (UniqueName: \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.236992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.236744 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.236992 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.236820 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.237713 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.237567 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.239744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.239713 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.239932 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.239913 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.257827 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.257797 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mtj\" (UniqueName: \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\") pod \"maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.275711 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.275678 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:55.358926 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.358883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:55.501584 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.500277 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:55.585149 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:55.585115 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:46:55.591254 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:46:55.591226 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5b7df7_1ca1_443d_9b95_5d5d9dced0ae.slice/crio-efb9c1b210987894da777865783b199384b3d24485252d3104d73c91337d01b0 WatchSource:0}: Error finding container efb9c1b210987894da777865783b199384b3d24485252d3104d73c91337d01b0: Status 404 returned error can't find the container with id efb9c1b210987894da777865783b199384b3d24485252d3104d73c91337d01b0 Oct 30 20:46:56.399587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:56.399547 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" event={"ID":"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae","Type":"ContainerStarted","Data":"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc"} Oct 30 20:46:56.400142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:56.399610 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" event={"ID":"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae","Type":"ContainerStarted","Data":"efb9c1b210987894da777865783b199384b3d24485252d3104d73c91337d01b0"} Oct 30 20:46:56.400142 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:56.399636 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:46:56.449806 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:56.449744 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" podStartSLOduration=2.115465756 podStartE2EDuration="2.449730176s" totalImagesPullingTime="334.26442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:56.447647432 +0000 UTC m=+4340.576881364" watchObservedRunningTime="2026-10-30 20:46:56.449730176 +0000 UTC m=+4340.578964088" Oct 30 20:46:56.876924 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:56.876833 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:58.494879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:46:58.494847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:00.105291 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:00.105255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:01.588851 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:01.588818 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:02.410772 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:02.410745 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:47:03.180857 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:03.180823 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:04.780876 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:04.780841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:06.402719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:06.402685 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:08.018360 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:08.018327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:08.499404 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:08.499373 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:09.606456 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:09.606424 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:11.107341 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:11.107308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:12.579640 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:12.579609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:14.086879 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:14.086847 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.482619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:15.482580 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.634739 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:15.634702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:16.065958 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:47:16.065917 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:47:17.070122 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:17.070058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:18.596979 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:18.596943 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:20.109238 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:20.109199 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:21.698248 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:21.698212 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:23.097789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:23.097709 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:24.037557 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:24.037518 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4"] Oct 30 20:47:24.048738 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:24.048706 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bksx4"] Oct 30 20:47:24.619224 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:24.619188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:25.994919 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:25.994880 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58862520-9f9c-4721-b908-8e30f67df475" path="/var/lib/kubelet/pods/58862520-9f9c-4721-b908-8e30f67df475/volumes" Oct 30 20:47:26.089619 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:26.089583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:27.584930 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:27.584898 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:29.137509 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:29.137452 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:30.584599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:30.584561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:32.118399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:32.118365 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:33.581540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:33.581478 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:35.176378 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:35.176344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:36.034256 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:36.034229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:36.567218 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:36.567181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:36.697013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:36.696983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:38.278268 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:38.278224 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:38.478603 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:38.478573 4959 scope.go:122] "RemoveContainer" containerID="19a66b047d8a18f241a603a6a72eb5ad95570667acb232b6933aa313531640e4" Oct 30 20:47:39.888183 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:39.888150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:41.381036 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:41.381004 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:42.999389 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:42.999359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:44.482994 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:44.482965 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:45.982854 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:45.982783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:46.069985 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:47:46.069951 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:47:47.582076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:47.582037 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:49.079949 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:49.079910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:50.542925 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:50.542890 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.998728 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:51.998693 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:53.586855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:53.586822 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:55.103138 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:55.103107 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:56.578765 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:56.578732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:57.989633 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:57.989600 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:59.495109 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:47:59.495059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:01.189519 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:01.189472 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:02.694141 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:02.694106 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:04.376795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:04.376763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:05.891721 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:05.891687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:07.382777 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:07.382736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:08.869047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:08.869006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:10.368972 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:10.368935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:11.865626 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:11.865589 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:13.439214 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:13.439115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:14.862709 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:14.862672 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:16.065244 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:48:16.065208 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:48:16.453614 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:16.453581 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:18.207014 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:18.206976 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:18.794013 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:18.793979 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:18.908540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:18.908487 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:19.730209 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:19.730167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:21.291281 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:21.291246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:22.868441 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:22.868396 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:24.628082 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:24.628045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:26.383161 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:26.383107 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:28.028305 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:28.028270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:29.764985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:29.764949 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:31.488453 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:31.488415 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:33.206886 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:33.206854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:34.885066 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:34.885003 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:36.162605 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:36.162570 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:36.411073 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:36.411043 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:38.086295 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:38.086261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:38.501913 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:38.501882 4959 scope.go:122] "RemoveContainer" containerID="39d061cef6baa36b76bdb20b86fcf280ace0e9fb92d7321a07e66bfe1e5f6f80" Oct 30 20:48:39.124895 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:39.124859 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:39.263577 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:39.263541 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:39.693523 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:39.693456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:46.065199 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:48:46.065162 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:48:46.088076 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.087976 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:48:46.088354 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.088287 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" podUID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" containerName="maas-api" containerID="cri-o://299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc" gracePeriod=30 Oct 30 20:48:46.148489 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.148456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:46.341047 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.341024 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:48:46.427083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.427033 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\" (UniqueName: \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\") pod \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " Oct 30 20:48:46.427083 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.427083 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\") pod \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " Oct 30 20:48:46.427301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.427198 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\") pod \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " Oct 30 20:48:46.427301 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.427298 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\") pod \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\" (UID: \"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae\") " Oct 30 20:48:46.427715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.427690 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca" pod "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" (UID: "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:48:46.429464 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.429441 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls" pod "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" (UID: "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:46.429888 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.429867 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs" pod "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" (UID: "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:46.429971 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.429911 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj" pod "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" (UID: "fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae"). InnerVolumeSpecName "kube-api-access-k2mtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:46.528780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.528723 4959 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:48:46.528780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.528773 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2mtj\" (UniqueName: \"kubernetes.io/projected/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-kube-api-access-k2mtj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:48:46.528780 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.528786 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-metrics-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:48:46.529117 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.528799 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Oct 30 20:48:46.667789 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.667747 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:46.785677 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.785643 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:46.929012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.928926 4959 generic.go:356] "Generic (PLEG): container finished" podID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" containerID="299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc" exitCode=0 Oct 30 20:48:46.929012 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.928974 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" Oct 30 20:48:46.929231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.929007 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" event={"ID":"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae","Type":"ContainerDied","Data":"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc"} Oct 30 20:48:46.929231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.929063 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj" event={"ID":"fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae","Type":"ContainerDied","Data":"efb9c1b210987894da777865783b199384b3d24485252d3104d73c91337d01b0"} Oct 30 20:48:46.929231 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.929085 4959 scope.go:122] "RemoveContainer" containerID="299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc" Oct 30 20:48:46.938855 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.938838 4959 scope.go:122] "RemoveContainer" containerID="299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc" Oct 30 20:48:46.939097 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:48:46.939082 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc\": container with ID starting with 299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc not found: ID does not exist" containerID="299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc" Oct 30 20:48:46.939148 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.939104 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc"} err="failed to get container status \"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc\": rpc error: code = NotFound desc = could not find container \"299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc\": container with ID starting with 299c110da2b0f455918f4283b388a758ac8411c80db68414bba7c15df6011bbc not found: ID does not exist" Oct 30 20:48:46.980624 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.980599 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:48:46.991334 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:46.991310 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-72e6e5-7d674d8b8d-x5qfj"] Oct 30 20:48:47.364385 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:47.364299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:47.995290 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:47.995256 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" path="/var/lib/kubelet/pods/fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae/volumes" Oct 30 20:48:54.224258 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:54.224217 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:54.340737 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:54.340702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:54.628572 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:48:54.628460 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:01.997304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:01.997265 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:06.031834 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:06.031677 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j"] Oct 30 20:49:06.041730 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:06.041698 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05491724-7p95j"] Oct 30 20:49:08.029516 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:08.028729 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d351315-1d65-4b68-878c-b59671b55247" path="/var/lib/kubelet/pods/0d351315-1d65-4b68-878c-b59671b55247/volumes" Oct 30 20:49:16.063958 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:49:16.063922 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:49:36.277916 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:36.277880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:36.874021 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:36.873986 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:49:36.877178 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:36.877151 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:49:46.065107 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:49:46.065068 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:49:51.430866 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:49:51.430830 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9874ff978-sr9zq_a0eaa649-8275-4d68-8e6a-3dd606034036/authorino/0.log" Oct 30 20:50:01.972374 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:01.972339 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-57ff5bcdf7-twvfd_5f103348-8709-416f-af44-07d3eee0737d/manager/0.log" Oct 30 20:50:04.641217 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:04.641184 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f6f44c6f-8k9rf_b4a0d58f-ce59-474f-85f2-e69fff014d24/maas-api/0.log" Oct 30 20:50:05.036469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:05.036384 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mszb2_67b5a730-c9ed-49dc-885c-86e1a5ee3b29/revoke-keys/0.log" Oct 30 20:50:05.592736 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:05.592704 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d3feede2-shkgw_9ed307e6-da08-4266-9c4a-abe836d1a5c1/revoke-keys/0.log" Oct 30 20:50:07.499715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:07.499685 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9874ff978-sr9zq_a0eaa649-8275-4d68-8e6a-3dd606034036/authorino/0.log" Oct 30 20:50:09.750540 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:09.750511 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vcqz9_11a13b5e-fa09-42e7-93fd-54108c02ac43/discovery/0.log" Oct 30 20:50:10.177118 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:10.177084 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-tc2wz_77496d52-d5c3-4e02-88f6-2036d4547a8c/kube-auth-proxy/0.log" Oct 30 20:50:11.168724 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:11.168695 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-c69zv_0c6676ac-0454-4b35-92b0-0969d06bb034/istio-proxy/0.log" Oct 30 20:50:11.356304 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:11.356273 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-h4twk_0f8bd228-4267-40d4-af2e-198251489fce/istio-proxy/0.log" Oct 30 20:50:11.546683 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:11.546612 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pq5nh_5ba7dd9e-9da3-4885-b01a-4a942d81c097/istio-proxy/0.log" Oct 30 20:50:12.149933 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:12.149904 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-ss8wt_98c950f1-eea3-4c70-9cda-3640eb311eee/istio-proxy/0.log" Oct 30 20:50:12.546277 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:12.546194 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7zvc2_561f2758-1a08-4cbb-b4cd-22080c44c16f/payload-pre-processing/0.log" Oct 30 20:50:12.925715 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:12.925684 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd9894b6f-skqbz_498b6873-9b05-4819-8d44-c6bbab466828/router/0.log" Oct 30 20:50:14.112005 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:14.111975 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5npp4_10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde/main/0.log" Oct 30 20:50:14.683544 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:14.683515 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65_890d2c73-01dc-453a-b0d6-9db8142c9980/main/0.log" Oct 30 20:50:16.064922 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:50:16.064888 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:50:16.211043 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:16.211017 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9874ff978-sr9zq_a0eaa649-8275-4d68-8e6a-3dd606034036/authorino/0.log" Oct 30 20:50:25.496379 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:25.496348 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-57ff5bcdf7-twvfd_5f103348-8709-416f-af44-07d3eee0737d/manager/0.log" Oct 30 20:50:27.388054 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:27.388021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:27.688985 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:27.688916 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76f6f44c6f-8k9rf_b4a0d58f-ce59-474f-85f2-e69fff014d24/maas-api/0.log" Oct 30 20:50:28.086077 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:28.086006 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mszb2_67b5a730-c9ed-49dc-885c-86e1a5ee3b29/revoke-keys/0.log" Oct 30 20:50:28.490062 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:28.490032 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d3feede2-shkgw_9ed307e6-da08-4266-9c4a-abe836d1a5c1/revoke-keys/0.log" Oct 30 20:50:30.016805 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:30.016776 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9874ff978-sr9zq_a0eaa649-8275-4d68-8e6a-3dd606034036/authorino/0.log" Oct 30 20:50:32.071744 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:32.071708 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vcqz9_11a13b5e-fa09-42e7-93fd-54108c02ac43/discovery/0.log" Oct 30 20:50:32.450080 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:32.450050 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-tc2wz_77496d52-d5c3-4e02-88f6-2036d4547a8c/kube-auth-proxy/0.log" Oct 30 20:50:33.448465 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:33.448435 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-c69zv_0c6676ac-0454-4b35-92b0-0969d06bb034/istio-proxy/0.log" Oct 30 20:50:33.648877 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:33.648839 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-h4twk_0f8bd228-4267-40d4-af2e-198251489fce/istio-proxy/0.log" Oct 30 20:50:33.838517 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:33.838419 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pq5nh_5ba7dd9e-9da3-4885-b01a-4a942d81c097/istio-proxy/0.log" Oct 30 20:50:34.431718 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:34.431688 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-ss8wt_98c950f1-eea3-4c70-9cda-3640eb311eee/istio-proxy/0.log" Oct 30 20:50:34.834558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:34.834461 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7zvc2_561f2758-1a08-4cbb-b4cd-22080c44c16f/payload-pre-processing/0.log" Oct 30 20:50:35.213302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:35.213266 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd9894b6f-skqbz_498b6873-9b05-4819-8d44-c6bbab466828/router/0.log" Oct 30 20:50:35.956144 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:35.956117 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5npp4_10a0dd10-7ce2-48e4-8fe6-e7c1b9019fde/main/0.log" Oct 30 20:50:36.408469 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:36.408437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:36.578650 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:36.578624 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8skr65_890d2c73-01dc-453a-b0d6-9db8142c9980/main/0.log" Oct 30 20:50:37.040382 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:37.040350 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2"] Oct 30 20:50:37.050941 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:37.050915 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mszb2"] Oct 30 20:50:37.994446 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:37.994413 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b5a730-c9ed-49dc-885c-86e1a5ee3b29" path="/var/lib/kubelet/pods/67b5a730-c9ed-49dc-885c-86e1a5ee3b29/volumes" Oct 30 20:50:38.532554 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:38.532521 4959 scope.go:122] "RemoveContainer" containerID="7d40dfd172bb4793f6ce8eb9835733983c692afd4920de158b71e9fe295249b8" Oct 30 20:50:38.540989 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:38.540968 4959 scope.go:122] "RemoveContainer" containerID="aef18a6adeb6764801c392868175ce7923776a782689b93f59b5154793f60a9b" Oct 30 20:50:46.064121 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:50:46.064082 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:50:57.038869 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:57.038830 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw"] Oct 30 20:50:57.050026 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:57.050001 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3feede2-shkgw"] Oct 30 20:50:57.995416 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:50:57.995385 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed307e6-da08-4266-9c4a-abe836d1a5c1" path="/var/lib/kubelet/pods/9ed307e6-da08-4266-9c4a-abe836d1a5c1/volumes" Oct 30 20:51:00.566064 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:00.566029 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:16.064031 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:51:16.063992 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:51:36.521513 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:36.521472 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:37.301448 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:37.301404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:38.565276 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:38.565243 4959 scope.go:122] "RemoveContainer" containerID="9f61aed494f9a5ba41c1fb987e5873b24afe58e968c6dc4952a01dcb2b0745b3" Oct 30 20:51:46.065086 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:51:46.065044 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:51:57.789812 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:57.789766 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:57.898275 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:51:57.898241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:02.489558 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:02.489518 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-shwzj_b001525d-fb68-4e3f-b967-f1a8d43f620b/global-pull-secret-syncer/0.log" Oct 30 20:52:02.843386 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:02.843309 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7t7rs_16eeabd2-8f23-411a-b951-6207f95bdb43/connection-checker/0.log" Oct 30 20:52:02.958719 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:02.958692 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-h4vsd_fc0ef185-0820-4ae9-955b-0f35510667d9/connection-checker/0.log" Oct 30 20:52:03.073815 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:03.073789 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kv78f_df601210-c5b1-4f3f-939b-bb8a11c3759d/connection-checker/0.log" Oct 30 20:52:03.223638 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:03.223605 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6bzkf_6c799558-73c8-45a2-83bf-dd99b24b8a36/konnectivity-agent/0.log" Oct 30 20:52:03.754068 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:03.754040 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-115.ec2.internal_4bb73836ec8ec4c4107421909743aa5c/haproxy/0.log" Oct 30 20:52:10.656902 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:10.656861 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9874ff978-sr9zq_a0eaa649-8275-4d68-8e6a-3dd606034036/authorino/0.log" Oct 30 20:52:12.458795 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.458759 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv"] Oct 30 20:52:12.459240 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.459223 4959 cpu_manager.go:414] "containerMap: removing container" podUID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" containerName="maas-api" Oct 30 20:52:12.459289 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.459243 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" containerName="maas-api" Oct 30 20:52:12.459412 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.459397 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5b7df7-1ca1-443d-9b95-5d5d9dced0ae" containerName="maas-api" Oct 30 20:52:12.462599 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.462571 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.481432 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.481410 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv"] Oct 30 20:52:12.565873 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.565846 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"openshift-service-ca.crt\"" Oct 30 20:52:12.566072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.565990 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7snw\"/\"default-dockercfg-965f6\"" Oct 30 20:52:12.566072 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.566028 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7snw\"/\"kube-root-ca.crt\"" Oct 30 20:52:12.577426 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.577405 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-proc\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.577598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.577434 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cwm\" (UniqueName: \"kubernetes.io/projected/81c71ff6-4a81-4187-bed4-4640a9787f67-kube-api-access-45cwm\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.577598 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.577465 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-sys\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.577727 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.577640 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-lib-modules\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.577798 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.577779 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-podres\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679075 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679040 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cwm\" (UniqueName: \"kubernetes.io/projected/81c71ff6-4a81-4187-bed4-4640a9787f67-kube-api-access-45cwm\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679098 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-sys\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679247 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679147 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-lib-modules\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679242 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-podres\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679258 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-sys\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679281 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-proc\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679333 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679325 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-podres\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679327 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-lib-modules\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.679524 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.679363 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81c71ff6-4a81-4187-bed4-4640a9787f67-proc\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.697302 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.697269 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cwm\" (UniqueName: \"kubernetes.io/projected/81c71ff6-4a81-4187-bed4-4640a9787f67-kube-api-access-45cwm\") pod \"perf-node-gather-daemonset-cw2sv\" (UID: \"81c71ff6-4a81-4187-bed4-4640a9787f67\") " pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:12.775573 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:12.775472 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:13.073963 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:13.073863 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv"] Oct 30 20:52:13.079943 ip-10-0-132-115 kubenswrapper[4959]: W1030 20:52:13.079915 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod81c71ff6_4a81_4187_bed4_4640a9787f67.slice/crio-9df4bb7e5918e109420ee3b60987a22bf816d830a103c45c941e71dea13d8cef WatchSource:0}: Error finding container 9df4bb7e5918e109420ee3b60987a22bf816d830a103c45c941e71dea13d8cef: Status 404 returned error can't find the container with id 9df4bb7e5918e109420ee3b60987a22bf816d830a103c45c941e71dea13d8cef Oct 30 20:52:13.909198 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:13.909164 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-5484c_3e7e7bcf-5f36-44d1-9232-167eebe939aa/cluster-monitoring-operator/0.log" Oct 30 20:52:13.993954 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:13.993927 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:13.994108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:13.993977 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" event={"ID":"81c71ff6-4a81-4187-bed4-4640a9787f67","Type":"ContainerStarted","Data":"0e8b1121c49ae83a29c5d559b41e0c84e58e61f9cd1c5d51a1f181788992fca3"} Oct 30 20:52:13.994108 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:13.993997 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" event={"ID":"81c71ff6-4a81-4187-bed4-4640a9787f67","Type":"ContainerStarted","Data":"9df4bb7e5918e109420ee3b60987a22bf816d830a103c45c941e71dea13d8cef"} Oct 30 20:52:14.027204 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:14.027154 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" podStartSLOduration=2.027142992 podStartE2EDuration="2.027142992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:14.026202353 +0000 UTC m=+4658.155436274" watchObservedRunningTime="2026-10-30 20:52:14.027142992 +0000 UTC m=+4658.156376967" Oct 30 20:52:15.366093 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:15.366059 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nc5gg_ff46f19b-f529-49cb-a027-c1b0d28295e1/node-exporter/0.log" Oct 30 20:52:15.486672 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:15.486644 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nc5gg_ff46f19b-f529-49cb-a027-c1b0d28295e1/kube-rbac-proxy/0.log" Oct 30 20:52:15.605017 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:15.604993 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nc5gg_ff46f19b-f529-49cb-a027-c1b0d28295e1/init-textfile/0.log" Oct 30 20:52:16.064445 ip-10-0-132-115 kubenswrapper[4959]: E1030 20:52:16.064414 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q8jlb" containerName="extractor" Oct 30 20:52:19.760399 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:19.760362 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-mb2fw_5e27c42b-524e-4f97-b89b-87047a1d32f7/networking-console-plugin/0.log" Oct 30 20:52:20.005621 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:20.005594 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b7snw/perf-node-gather-daemonset-cw2sv" Oct 30 20:52:20.935034 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:20.935004 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/2.log" Oct 30 20:52:20.939225 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:20.939205 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zl8sn_73261ca6-68ed-4f8d-bc9e-f4eb802b75a4/console-operator/3.log" Oct 30 20:52:23.497870 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:23.497845 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ppsxz_3da0ba8c-d364-4754-9069-47fe4a2017a6/volume-data-source-validator/0.log" Oct 30 20:52:26.471928 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:26.471894 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sg9nh_3d7a9e09-ccd3-4c01-92fd-41844e62b60f/dns/0.log" Oct 30 20:52:26.596563 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:26.596534 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sg9nh_3d7a9e09-ccd3-4c01-92fd-41844e62b60f/kube-rbac-proxy/0.log" Oct 30 20:52:26.843415 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:26.843340 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jw9s8_7cde5a04-1c75-44b3-8e49-bda52bb78768/dns-node-resolver/0.log" Oct 30 20:52:28.223052 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:28.223024 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fr4xc_04a17b98-bb85-4bde-b1b4-29d1b39ad1c5/node-ca/0.log" Oct 30 20:52:28.707974 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:28.707937 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vcqz9_11a13b5e-fa09-42e7-93fd-54108c02ac43/discovery/0.log" Oct 30 20:52:28.808126 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:28.808086 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66b985bdbd-tc2wz_77496d52-d5c3-4e02-88f6-2036d4547a8c/kube-auth-proxy/0.log" Oct 30 20:52:29.123944 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.123858 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-c69zv_0c6676ac-0454-4b35-92b0-0969d06bb034/istio-proxy/0.log" Oct 30 20:52:29.194074 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.194041 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-h4twk_0f8bd228-4267-40d4-af2e-198251489fce/istio-proxy/0.log" Oct 30 20:52:29.268285 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.268260 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pq5nh_5ba7dd9e-9da3-4885-b01a-4a942d81c097/istio-proxy/0.log" Oct 30 20:52:29.477848 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.477821 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-ss8wt_98c950f1-eea3-4c70-9cda-3640eb311eee/istio-proxy/0.log" Oct 30 20:52:29.608587 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.608559 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7zvc2_561f2758-1a08-4cbb-b4cd-22080c44c16f/payload-pre-processing/0.log" Oct 30 20:52:29.748608 ip-10-0-132-115 kubenswrapper[4959]: I1030 20:52:29.748518 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-fd9894b6f-skqbz_498b6873-9b05-4819-8d44-c6bbab466828/router/0.log"